2023-09-23T06:56:47.819 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-23T06:56:47.819 DEBUG:teuthology.run:Teuthology command: teuthology --name rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi --owner scheduled_rraja@teuthology --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} --verbose --archive /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649 -- /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/orig.config.yaml 2023-09-23T06:56:47.854 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-23T06:56:47.932 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649 branch: wip-62891 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7401649' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-62891 ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 enable experimental unrecoverable data corrupting features: '*' mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug randomize hobject sort order: false osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-62891 sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2241 sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-62891 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e targets: smithi140.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM7EfadN9AifUj3gzNG/vZ54NYmDnMRc27pZKNKLHzZBLjbo6PApZOFYWXCd3JzsDiy0T0vb/YQkmWSvAIr63Vw= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2 - sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile - sudo ceph osd pool set datapool allow_ec_overwrites true - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-23_06:37:41 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-23T06:56:47.932 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa; will attempt to use it 2023-09-23T06:56:47.933 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks 2023-09-23T06:56:47.933 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-23T06:56:47.935 INFO:teuthology.task.internal:Checking packages... 2023-09-23T06:56:47.958 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e' 2023-09-23T06:56:47.958 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-23T06:56:47.959 INFO:teuthology.packaging:ref: None 2023-09-23T06:56:47.959 INFO:teuthology.packaging:tag: None 2023-09-23T06:56:47.959 INFO:teuthology.packaging:branch: wip-62891 2023-09-23T06:56:47.959 INFO:teuthology.packaging:sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T06:56:47.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-62891 2023-09-23T06:56:48.201 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6334.gee696c23 2023-09-23T06:56:48.201 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-23T06:56:48.209 INFO:teuthology.task.internal:no buildpackages task found 2023-09-23T06:56:48.209 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-23T06:56:48.242 INFO:teuthology.task.internal:Saving configuration 2023-09-23T06:56:48.254 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-23T06:56:48.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-23T06:56:48.298 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-23 06:51:25.593899', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM7EfadN9AifUj3gzNG/vZ54NYmDnMRc27pZKNKLHzZBLjbo6PApZOFYWXCd3JzsDiy0T0vb/YQkmWSvAIr63Vw='} 2023-09-23T06:56:48.298 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-23T06:56:48.304 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-23T06:56:48.304 INFO:teuthology.run_tasks:Running task console_log... 2023-09-23T06:56:48.364 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe96dfdcaf0>, signals=[15]) 2023-09-23T06:56:48.364 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-23T06:56:48.396 INFO:teuthology.task.internal:Opening connections... 2023-09-23T06:56:48.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T06:56:48.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T06:56:48.464 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-23T06:56:48.471 DEBUG:teuthology.orchestra.run.smithi140:> uname -m 2023-09-23T06:56:48.535 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64 2023-09-23T06:56:48.536 DEBUG:teuthology.orchestra.run.smithi140:> cat /etc/os-release 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Stream" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31" 2023-09-23T06:56:48.597 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-23T06:56:48.598 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://centos.org/" 2023-09-23T06:56:48.598 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-23T06:56:48.598 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-23T06:56:48.598 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-23T06:56:48.600 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-09-23T06:56:48.617 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-23T06:56:48.625 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-23T06:56:48.650 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-23T06:56:48.650 DEBUG:teuthology.orchestra.run.smithi140:> test '!' -e /home/ubuntu/cephtest 2023-09-23T06:56:48.669 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-23T06:56:48.675 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-23T06:56:48.675 DEBUG:teuthology.orchestra.run.smithi140:> test -z $(ls -A /var/lib/ceph) 2023-09-23T06:56:48.738 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-23T06:56:48.770 INFO:teuthology.run_tasks:Running task kernel... 2023-09-23T06:56:48.784 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-23T06:56:48.784 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-23T06:56:48.784 DEBUG:teuthology.orchestra.run.smithi140:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-23T06:56:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T06:56:48.806 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2023-09-23T06:56:48.867 INFO:teuthology.orchestra.run.smithi140.stdout:4.18.0-489.el8.x86_64 2023-09-23T06:56:48.868 INFO:teuthology.task.kernel:Running kernel on smithi140: 4.18.0-489.el8.x86_64 2023-09-23T06:56:48.868 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum install -y kernel 2023-09-23T06:56:52.557 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - AppStream 31 MB/s | 33 MB 00:01 2023-09-23T06:57:00.199 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 47 MB 00:01 2023-09-23T06:57:08.731 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2023-09-23T06:57:09.457 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-23T06:57:09.750 INFO:teuthology.orchestra.run.smithi140.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-09-23T06:57:11.737 INFO:teuthology.orchestra.run.smithi140.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2023-09-23T06:57:15.609 INFO:teuthology.orchestra.run.smithi140.stdout:lab-extras 433 kB/s | 24 kB 00:00 2023-09-23T06:57:19.100 INFO:teuthology.orchestra.run.smithi140.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-23T06:57:19.100 INFO:teuthology.orchestra.run.smithi140.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-23T06:57:19.339 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T06:57:19.340 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T06:57:19.340 INFO:teuthology.orchestra.run.smithi140.stdout: Package Architecture Version Repository Size 2023-09-23T06:57:19.340 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T06:57:19.340 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout: kernel x86_64 4.18.0-513.el8 baseos 10 M 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-core x86_64 4.18.0-513.el8 baseos 43 M 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-modules x86_64 4.18.0-513.el8 baseos 35 M 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout:Install 3 Packages 2023-09-23T06:57:19.341 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:19.342 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 88 M 2023-09-23T06:57:19.342 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 96 M 2023-09-23T06:57:19.342 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T06:57:20.378 INFO:teuthology.orchestra.run.smithi140.stdout:(1/3): kernel-4.18.0-513.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2023-09-23T06:57:21.177 INFO:teuthology.orchestra.run.smithi140.stdout:(2/3): kernel-core-4.18.0-513.el8.x86_64.rpm 26 MB/s | 43 MB 00:01 2023-09-23T06:57:21.719 INFO:teuthology.orchestra.run.smithi140.stdout:(3/3): kernel-modules-4.18.0-513.el8.x86_64.rpm 16 MB/s | 35 MB 00:02 2023-09-23T06:57:21.720 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T06:57:21.720 INFO:teuthology.orchestra.run.smithi140.stdout:Total 37 MB/s | 88 MB 00:02 2023-09-23T06:57:22.398 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T06:57:22.832 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T06:57:22.832 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T06:57:24.038 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T06:57:24.056 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T06:57:26.923 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T06:57:27.515 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-23T06:57:31.147 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-23T06:57:31.548 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-23T06:57:34.988 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-23T06:57:35.252 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-4.18.0-513.el8.x86_64 3/3 2023-09-23T06:57:57.039 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 3/3 2023-09-23T06:57:57.062 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-23T06:57:57.410 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-4.18.0-513.el8.x86_64 3/3 2023-09-23T06:57:57.410 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : kernel-4.18.0-513.el8.x86_64 1/3 2023-09-23T06:57:57.410 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : kernel-core-4.18.0-513.el8.x86_64 2/3 2023-09-23T06:57:58.122 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-4.18.0-513.el8.x86_64 kernel-core-4.18.0-513.el8.x86_64 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-modules-4.18.0-513.el8.x86_64 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:58.123 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T06:57:58.279 DEBUG:teuthology.orchestra.run.smithi140:> rpm -q kernel | sort -rV | head -n 1 2023-09-23T06:57:58.499 INFO:teuthology.orchestra.run.smithi140.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-23T06:57:58.499 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-23T06:57:58.499 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-513.el8.x86_64 2023-09-23T06:57:58.499 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-23T06:57:58.499 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-23T06:57:58.499 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-23T06:57:58.499 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi140.front.sepia.ceph.com, path=None, version=distro) 2023-09-23T06:57:58.500 DEBUG:teuthology.orchestra.run.smithi140:> rpm -q kernel | sort -rV | head -n 1 2023-09-23T06:57:58.708 INFO:teuthology.orchestra.run.smithi140.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-23T06:57:58.708 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-23T06:57:58.708 DEBUG:teuthology.orchestra.run.smithi140:> sudo rpm -qi grub2-tools 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Name : grub2-tools 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Epoch : 1 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Version : 2.02 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Release : 129.el8 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Architecture: x86_64 2023-09-23T06:57:58.742 INFO:teuthology.orchestra.run.smithi140.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Group : System Environment/Base 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Size : 9265586 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:License : GPLv3+ 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Relocations : (not relocatable) 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Packager : CentOS BuildSystem 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Vendor : CentOS 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-23T06:57:58.743 INFO:teuthology.orchestra.run.smithi140.stdout:Summary : Support tools for GRUB. 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:Description : 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:hardware devices. 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T06:57:58.744 INFO:teuthology.orchestra.run.smithi140.stdout:This subpackage provides tools for support of all platforms. 2023-09-23T06:57:58.745 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-23T06:57:58.745 INFO:teuthology.task.kernel:Updating grub on smithi140 to boot 4.18.0-513.el8.x86_64 2023-09-23T06:57:58.745 DEBUG:teuthology.orchestra.run.smithi140:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-23T06:58:00.176 INFO:teuthology.orchestra.run.smithi140.stderr:Generating grub configuration file ... 2023-09-23T06:58:02.336 INFO:teuthology.orchestra.run.smithi140.stderr:done 2023-09-23T06:58:02.337 DEBUG:teuthology.orchestra.run.smithi140:> mktemp 2023-09-23T06:58:02.370 INFO:teuthology.orchestra.run.smithi140.stdout:/tmp/tmp.a4lM4rOahv 2023-09-23T06:58:02.370 DEBUG:teuthology.orchestra.run.smithi140:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.a4lM4rOahv 2023-09-23T06:58:02.447 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0666 /tmp/tmp.a4lM4rOahv 2023-09-23T06:58:02.590 DEBUG:teuthology.orchestra.remote:smithi140:/tmp/tmp.a4lM4rOahv is 6KB 2023-09-23T06:58:02.638 DEBUG:teuthology.orchestra.run.smithi140:> rm -fr /tmp/tmp.a4lM4rOahv 2023-09-23T06:58:02.655 DEBUG:teuthology.orchestra.run.smithi140:> sudo /bin/ls /boot/loader/entries || true 2023-09-23T06:58:02.721 INFO:teuthology.orchestra.run.smithi140.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-23T06:58:02.721 INFO:teuthology.orchestra.run.smithi140.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-23T06:58:02.721 INFO:teuthology.orchestra.run.smithi140.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-23T06:58:02.721 INFO:teuthology.orchestra.run.smithi140.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64.conf 2023-09-23T06:58:02.722 DEBUG:teuthology.orchestra.run.smithi140:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64 2023-09-23T06:58:02.887 DEBUG:teuthology.orchestra.run.smithi140:> sudo shutdown -r now 2023-09-23T06:58:32.911 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-23T06:58:32.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T06:58:32.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T06:59:07.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-09-23T06:59:16.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T06:59:16.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T07:00:16.572 DEBUG:teuthology.orchestra.remote:timed out 2023-09-23T07:00:28.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T07:00:28.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T07:00:28.897 DEBUG:teuthology.orchestra.run.smithi140:> true 2023-09-23T07:00:29.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T07:00:29.207 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-513.el8.x86_64"... 2023-09-23T07:00:29.208 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2023-09-23T07:00:29.226 INFO:teuthology.orchestra.run.smithi140.stdout:4.18.0-513.el8.x86_64 2023-09-23T07:00:29.227 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.el8.x86_64 vs 4.18.0-513.el8.x86_64 2023-09-23T07:00:29.227 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-23T07:00:29.227 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-23T07:00:30.228 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-23T07:00:30.228 DEBUG:teuthology.orchestra.run.smithi140:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-23T07:00:30.299 INFO:teuthology.orchestra.run.smithi140.stdout:ttyS1 2023-09-23T07:00:30.313 DEBUG:teuthology.parallel:result is None 2023-09-23T07:00:30.313 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-23T07:00:30.320 INFO:teuthology.task.internal:Creating test directory... 2023-09-23T07:00:30.320 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-23T07:00:30.341 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-23T07:00:30.367 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-23T07:00:30.400 INFO:teuthology.task.internal:Creating archive directory... 2023-09-23T07:00:30.400 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-23T07:00:30.429 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-23T07:00:30.441 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-23T07:00:30.442 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-23T07:00:30.501 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-23T07:00:30.512 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-23T07:00:30.513 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-23T07:00:30.521 INFO:teuthology.task.internal:Configuring sudo... 2023-09-23T07:00:30.521 DEBUG:teuthology.orchestra.run.smithi140:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-23T07:00:30.582 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-23T07:00:30.591 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-23T07:00:30.592 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-23T07:00:30.640 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-23T07:00:30.726 DEBUG:teuthology.orchestra.run.smithi140:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-23T07:00:30.775 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-23T07:00:30.859 DEBUG:teuthology.orchestra.run.smithi140:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-23T07:00:30.902 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:00:30.902 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-23T07:00:30.970 DEBUG:teuthology.orchestra.run.smithi140:> sudo service rsyslog restart 2023-09-23T07:00:31.069 INFO:teuthology.orchestra.run.smithi140.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-23T07:00:31.337 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-23T07:00:31.345 INFO:teuthology.task.internal:Starting timer... 2023-09-23T07:00:31.345 INFO:teuthology.run_tasks:Running task pcp... 2023-09-23T07:00:31.386 INFO:teuthology.run_tasks:Running task selinux... 2023-09-23T07:00:31.419 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-23T07:00:31.419 DEBUG:teuthology.orchestra.run.smithi140:> sudo service auditd rotate 2023-09-23T07:00:31.506 INFO:teuthology.orchestra.run.smithi140.stdout:Rotating logs: 2023-09-23T07:00:31.507 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-23T07:00:31.507 DEBUG:teuthology.orchestra.run.smithi140:> /usr/sbin/getenforce 2023-09-23T07:00:31.550 INFO:teuthology.orchestra.run.smithi140.stdout:Permissive 2023-09-23T07:00:31.551 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi140.front.sepia.ceph.com': 'permissive'} 2023-09-23T07:00:31.551 DEBUG:teuthology.orchestra.run.smithi140:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-23T07:00:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T07:00:31.619 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-23T07:00:31.619 DEBUG:teuthology.orchestra.run.smithi140:> sudo /usr/sbin/setenforce permissive 2023-09-23T07:00:31.686 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-23T07:00:31.696 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-23T07:00:31.696 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-23T07:00:31.712 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-23T07:00:31.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi140.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-23T07:07:47.292 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi140.front.sepia.ceph.com')] 2023-09-23T07:07:47.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T07:07:47.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T07:07:47.362 DEBUG:teuthology.orchestra.run.smithi140:> true 2023-09-23T07:07:47.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-09-23T07:07:47.390 INFO:teuthology.run_tasks:Running task clock... 2023-09-23T07:07:47.400 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-23T07:07:47.400 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-23T07:07:47.400 DEBUG:teuthology.orchestra.run.smithi140:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-23T07:07:47.462 INFO:teuthology.orchestra.run.smithi140.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-23T07:07:47.476 INFO:teuthology.orchestra.run.smithi140.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-23T07:07:47.551 INFO:teuthology.orchestra.run.smithi140.stderr:sudo: ntpd: command not found 2023-09-23T07:07:47.562 INFO:teuthology.orchestra.run.smithi140.stdout:506 Cannot talk to daemon 2023-09-23T07:07:47.577 INFO:teuthology.orchestra.run.smithi140.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-23T07:07:47.590 INFO:teuthology.orchestra.run.smithi140.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-23T07:07:47.661 INFO:teuthology.orchestra.run.smithi140.stderr:bash: ntpq: command not found 2023-09-23T07:07:47.664 INFO:teuthology.orchestra.run.smithi140.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-23T07:07:47.664 INFO:teuthology.orchestra.run.smithi140.stdout:=============================================================================== 2023-09-23T07:07:47.664 INFO:teuthology.orchestra.run.smithi140.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-23T07:07:47.664 INFO:teuthology.orchestra.run.smithi140.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-23T07:07:47.665 INFO:teuthology.orchestra.run.smithi140.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-23T07:07:47.665 INFO:teuthology.orchestra.run.smithi140.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-23T07:07:47.666 INFO:teuthology.run_tasks:Running task install... 2023-09-23T07:07:47.674 DEBUG:teuthology.task.install:project ceph 2023-09-23T07:07:47.674 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}} 2023-09-23T07:07:47.674 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'} 2023-09-23T07:07:47.674 INFO:teuthology.task.install:Using flavor: default 2023-09-23T07:07:47.680 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-23T07:07:47.680 INFO:teuthology.task.install:extra packages: [] 2023-09-23T07:07:47.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'tag': None, 'wait_for_package': False} 2023-09-23T07:07:47.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:07:47.997 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/ 2023-09-23T07:07:47.998 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6334.gee696c23 2023-09-23T07:07:48.105 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-23T07:07:48.106 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:07:48.106 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-23T07:07:48.135 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-23T07:07:48.135 DEBUG:teuthology.orchestra.run.smithi140:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-23T07:07:48.204 DEBUG:teuthology.orchestra.run.smithi140:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-23T07:07:48.282 DEBUG:teuthology.orchestra.run.smithi140:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-23T07:07:48.349 INFO:teuthology.orchestra.run.smithi140.stdout:check_obsoletes = 1 2023-09-23T07:07:48.350 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum clean all 2023-09-23T07:07:48.625 INFO:teuthology.orchestra.run.smithi140.stdout:76 files removed 2023-09-23T07:07:48.653 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-radosgw 2023-09-23T07:07:49.357 INFO:teuthology.orchestra.run.smithi140.stdout:ceph packages for x86_64 273 kB/s | 81 kB 00:00 2023-09-23T07:07:49.796 INFO:teuthology.orchestra.run.smithi140.stdout:ceph noarch packages 24 kB/s | 9.4 kB 00:00 2023-09-23T07:07:50.375 INFO:teuthology.orchestra.run.smithi140.stdout:ceph source packages 2.7 kB/s | 1.5 kB 00:00 2023-09-23T07:07:52.000 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - AppStream 21 MB/s | 33 MB 00:01 2023-09-23T07:08:01.233 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 47 MB 00:03 2023-09-23T07:08:09.025 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras 17 kB/s | 18 kB 00:01 2023-09-23T07:08:10.868 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras common packages 4.2 kB/s | 6.9 kB 00:01 2023-09-23T07:08:15.734 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - PowerTools 1.4 MB/s | 6.5 MB 00:04 2023-09-23T07:08:17.195 INFO:teuthology.orchestra.run.smithi140.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-23T07:08:19.340 INFO:teuthology.orchestra.run.smithi140.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 16 MB 00:01 2023-09-23T07:08:23.137 INFO:teuthology.orchestra.run.smithi140.stdout:lab-extras 418 kB/s | 24 kB 00:00 2023-09-23T07:08:27.572 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:08:27.574 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-radosgw x86_64 2:18.0.0-6334.gee696c23.el8 ceph 14 M 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout:Upgrading: 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: librados2 x86_64 2:18.0.0-6334.gee696c23.el8 ceph 3.7 M 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: librbd1 x86_64 2:18.0.0-6334.gee696c23.el8 ceph 3.7 M 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-base x86_64 2:18.0.0-6334.gee696c23.el8 ceph 5.4 M 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-common x86_64 2:18.0.0-6334.gee696c23.el8 ceph 23 M 2023-09-23T07:08:27.575 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-selinux x86_64 2:18.0.0-6334.gee696c23.el8 ceph 24 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: libcephfs2 x86_64 2:18.0.0-6334.gee696c23.el8 ceph 832 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: libradosstriper1 x86_64 2:18.0.0-6334.gee696c23.el8 ceph 522 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: librgw2 x86_64 2:18.0.0-6334.gee696c23.el8 ceph 6.9 M 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-argparse x86_64 2:18.0.0-6334.gee696c23.el8 ceph 45 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-common x86_64 2:18.0.0-6334.gee696c23.el8 ceph 125 k 2023-09-23T07:08:27.576 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cephfs x86_64 2:18.0.0-6334.gee696c23.el8 ceph 197 k 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rados x86_64 2:18.0.0-6334.gee696c23.el8 ceph 372 k 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rbd x86_64 2:18.0.0-6334.gee696c23.el8 ceph 364 k 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rgw x86_64 2:18.0.0-6334.gee696c23.el8 ceph 107 k 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout:Install 21 Packages 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout:Upgrade 2 Packages 2023-09-23T07:08:27.577 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:08:27.578 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 62 M 2023-09-23T07:08:27.578 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:08:30.238 INFO:teuthology.orchestra.run.smithi140.stdout:(1/23): ceph-base-18.0.0-6334.gee696c23.el8.x86 2.9 MB/s | 5.4 MB 00:01 2023-09-23T07:08:30.305 INFO:teuthology.orchestra.run.smithi140.stdout:(2/23): ceph-selinux-18.0.0-6334.gee696c23.el8. 367 kB/s | 24 kB 00:00 2023-09-23T07:08:30.480 INFO:teuthology.orchestra.run.smithi140.stdout:(3/23): libcephfs2-18.0.0-6334.gee696c23.el8.x8 4.7 MB/s | 832 kB 00:00 2023-09-23T07:08:30.654 INFO:teuthology.orchestra.run.smithi140.stdout:(4/23): libradosstriper1-18.0.0-6334.gee696c23. 2.9 MB/s | 522 kB 00:00 2023-09-23T07:08:31.290 INFO:teuthology.orchestra.run.smithi140.stdout:(5/23): ceph-radosgw-18.0.0-6334.gee696c23.el8. 4.7 MB/s | 14 MB 00:02 2023-09-23T07:08:31.390 INFO:teuthology.orchestra.run.smithi140.stdout:(6/23): python3-ceph-argparse-18.0.0-6334.gee69 448 kB/s | 45 kB 00:00 2023-09-23T07:08:31.490 INFO:teuthology.orchestra.run.smithi140.stdout:(7/23): python3-ceph-common-18.0.0-6334.gee696c 1.2 MB/s | 125 kB 00:00 2023-09-23T07:08:31.649 INFO:teuthology.orchestra.run.smithi140.stdout:(8/23): librgw2-18.0.0-6334.gee696c23.el8.x86_6 7.0 MB/s | 6.9 MB 00:00 2023-09-23T07:08:31.674 INFO:teuthology.orchestra.run.smithi140.stdout:(9/23): python3-cephfs-18.0.0-6334.gee696c23.el 1.0 MB/s | 197 kB 00:00 2023-09-23T07:08:31.933 INFO:teuthology.orchestra.run.smithi140.stdout:(10/23): ceph-common-18.0.0-6334.gee696c23.el8. 6.3 MB/s | 23 MB 00:03 2023-09-23T07:08:31.958 INFO:teuthology.orchestra.run.smithi140.stdout:(11/23): python3-rbd-18.0.0-6334.gee696c23.el8. 1.3 MB/s | 364 kB 00:00 2023-09-23T07:08:31.983 INFO:teuthology.orchestra.run.smithi140.stdout:(12/23): python3-rados-18.0.0-6334.gee696c23.el 1.1 MB/s | 372 kB 00:00 2023-09-23T07:08:32.009 INFO:teuthology.orchestra.run.smithi140.stdout:(13/23): python3-rgw-18.0.0-6334.gee696c23.el8. 1.4 MB/s | 107 kB 00:00 2023-09-23T07:08:32.185 INFO:teuthology.orchestra.run.smithi140.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 250 kB/s | 44 kB 00:00 2023-09-23T07:08:32.243 INFO:teuthology.orchestra.run.smithi140.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-09-23T07:08:32.268 INFO:teuthology.orchestra.run.smithi140.stdout:(16/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 MB/s | 354 kB 00:00 2023-09-23T07:08:32.402 INFO:teuthology.orchestra.run.smithi140.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 216 kB/s | 47 kB 00:00 2023-09-23T07:08:32.436 INFO:teuthology.orchestra.run.smithi140.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 352 kB/s | 59 kB 00:00 2023-09-23T07:08:32.577 INFO:teuthology.orchestra.run.smithi140.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 917 kB/s | 306 kB 00:00 2023-09-23T07:08:32.628 INFO:teuthology.orchestra.run.smithi140.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 333 kB/s | 75 kB 00:00 2023-09-23T07:08:32.851 INFO:teuthology.orchestra.run.smithi140.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2023-09-23T07:08:33.470 INFO:teuthology.orchestra.run.smithi140.stdout:(22/23): librbd1-18.0.0-6334.gee696c23.el8.x86_ 4.4 MB/s | 3.7 MB 00:00 2023-09-23T07:08:33.703 INFO:teuthology.orchestra.run.smithi140.stdout:(23/23): librados2-18.0.0-6334.gee696c23.el8.x8 3.3 MB/s | 3.7 MB 00:01 2023-09-23T07:08:33.705 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:08:33.706 INFO:teuthology.orchestra.run.smithi140.stdout:Total 10 MB/s | 62 MB 00:06 2023-09-23T07:08:33.749 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:08:33.795 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:08:33.795 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:08:34.169 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:08:34.172 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:08:35.503 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:08:35.727 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-09-23T07:08:35.973 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-09-23T07:08:36.152 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-09-23T07:08:36.555 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-09-23T07:08:36.777 INFO:teuthology.orchestra.run.smithi140.stdout: Upgrading : librados2-2:18.0.0-6334.gee696c23.el8.x86_64 4/25 2023-09-23T07:08:36.912 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librados2-2:18.0.0-6334.gee696c23.el8.x86_64 4/25 2023-09-23T07:08:37.221 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-rados-2:18.0.0-6334.gee696c23.el8.x86_64 5/25 2023-09-23T07:08:37.419 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 6/25 2023-09-23T07:08:37.827 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 6/25 2023-09-23T07:08:38.060 INFO:teuthology.orchestra.run.smithi140.stdout: Upgrading : librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 7/25 2023-09-23T07:08:38.118 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 7/25 2023-09-23T07:08:38.358 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-23T07:08:38.568 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-23T07:08:38.621 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-23T07:08:38.822 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-ceph-argparse-2:18.0.0-6334.gee696c23.el8. 10/25 2023-09-23T07:08:39.068 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cephfs-2:18.0.0-6334.gee696c23.el8.x86_64 11/25 2023-09-23T07:08:39.324 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-rbd-2:18.0.0-6334.gee696c23.el8.x86_64 12/25 2023-09-23T07:08:39.510 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libradosstriper1-2:18.0.0-6334.gee696c23.el8.x86_6 13/25 2023-09-23T07:08:39.573 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6334.gee696c23.el8.x86_6 13/25 2023-09-23T07:08:39.829 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-09-23T07:08:40.631 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-09-23T07:08:40.760 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : librgw2-2:18.0.0-6334.gee696c23.el8.x86_64 16/25 2023-09-23T07:08:40.835 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librgw2-2:18.0.0-6334.gee696c23.el8.x86_64 16/25 2023-09-23T07:08:41.035 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-rgw-2:18.0.0-6334.gee696c23.el8.x86_64 17/25 2023-09-23T07:08:41.245 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-23T07:08:41.711 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-ceph-common-2:18.0.0-6334.gee696c23.el8.x8 19/25 2023-09-23T07:08:44.581 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 20/25 2023-09-23T07:08:44.810 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 20/25 2023-09-23T07:08:45.428 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 20/25 2023-09-23T07:08:45.736 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 21/25 2023-09-23T07:08:45.872 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 21/25 2023-09-23T07:08:46.071 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-selinux-2:18.0.0-6334.gee696c23.el8.x86_64 22/25 2023-09-23T07:09:04.769 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6334.gee696c23.el8.x86_64 22/25 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /sys 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /proc 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /dev 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /run 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /mnt 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /var/tmp 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /home 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /tmp 2023-09-23T07:09:04.770 INFO:teuthology.orchestra.run.smithi140.stdout:skipping the directory /dev 2023-09-23T07:09:04.771 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:06.216 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-radosgw-2:18.0.0-6334.gee696c23.el8.x86_64 23/25 2023-09-23T07:09:06.238 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6334.gee696c23.el8.x86_64 23/25 2023-09-23T07:09:06.238 INFO:teuthology.orchestra.run.smithi140.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-23T07:09:06.450 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-23T07:09:06.450 INFO:teuthology.orchestra.run.smithi140.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 1/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 2/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-radosgw-2:18.0.0-6334.gee696c23.el8.x86_64 3/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-selinux-2:18.0.0-6334.gee696c23.el8.x86_64 4/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 5/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libradosstriper1-2:18.0.0-6334.gee696c23.el8.x86_6 6/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librgw2-2:18.0.0-6334.gee696c23.el8.x86_64 7/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6334.gee696c23.el8. 8/25 2023-09-23T07:09:07.163 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-ceph-common-2:18.0.0-6334.gee696c23.el8.x8 9/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cephfs-2:18.0.0-6334.gee696c23.el8.x86_64 10/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-rados-2:18.0.0-6334.gee696c23.el8.x86_64 11/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-rbd-2:18.0.0-6334.gee696c23.el8.x86_64 12/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-rgw-2:18.0.0-6334.gee696c23.el8.x86_64 13/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-23T07:09:07.164 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-23T07:09:07.165 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librados2-2:18.0.0-6334.gee696c23.el8.x86_64 22/25 2023-09-23T07:09:07.165 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-23T07:09:07.165 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 24/25 2023-09-23T07:09:08.177 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-23T07:09:08.177 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:08.178 INFO:teuthology.orchestra.run.smithi140.stdout:Upgraded: 2023-09-23T07:09:08.178 INFO:teuthology.orchestra.run.smithi140.stdout: librados2-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.178 INFO:teuthology.orchestra.run.smithi140.stdout: librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.178 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:09:08.178 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-radosgw-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-selinux-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-23T07:09:08.179 INFO:teuthology.orchestra.run.smithi140.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: libradosstriper1-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: librgw2-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-argparse-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.180 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-common-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cephfs-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rados-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rbd-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rgw-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:08.181 INFO:teuthology.orchestra.run.smithi140.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-23T07:09:08.182 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:08.182 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:08.488 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-test 2023-09-23T07:09:09.334 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:00:46 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:10.918 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-test x86_64 2:18.0.0-6334.gee696c23.el8 ceph 49 M 2023-09-23T07:09:10.919 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: libcephsqlite x86_64 2:18.0.0-6334.gee696c23.el8 ceph 184 k 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout:Install 6 Packages 2023-09-23T07:09:10.920 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:10.921 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 50 M 2023-09-23T07:09:10.921 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 222 M 2023-09-23T07:09:10.921 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:09:12.339 INFO:teuthology.orchestra.run.smithi140.stdout:(1/6): libcephsqlite-18.0.0-6334.gee696c23.el8. 854 kB/s | 184 kB 00:00 2023-09-23T07:09:12.374 INFO:teuthology.orchestra.run.smithi140.stdout:(2/6): jq-1.6-7.el8.x86_64.rpm 808 kB/s | 202 kB 00:00 2023-09-23T07:09:12.491 INFO:teuthology.orchestra.run.smithi140.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-09-23T07:09:12.584 INFO:teuthology.orchestra.run.smithi140.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 761 kB/s | 71 kB 00:00 2023-09-23T07:09:12.617 INFO:teuthology.orchestra.run.smithi140.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 674 kB/s | 187 kB 00:00 2023-09-23T07:09:14.725 INFO:teuthology.orchestra.run.smithi140.stdout:(6/6): ceph-test-18.0.0-6334.gee696c23.el8.x86_ 19 MB/s | 49 MB 00:02 2023-09-23T07:09:14.728 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:09:14.728 INFO:teuthology.orchestra.run.smithi140.stdout:Total 13 MB/s | 50 MB 00:03 2023-09-23T07:09:14.772 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:09:14.793 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:09:14.794 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:09:15.103 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:09:15.107 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:09:15.605 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:09:15.822 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-23T07:09:16.083 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-23T07:09:16.212 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-23T07:09:16.330 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-23T07:09:16.489 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-23T07:09:16.646 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libcephsqlite-2:18.0.0-6334.gee696c23.el8.x86_64 5/6 2023-09-23T07:09:21.382 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6334.gee696c23.el8.x86_64 5/6 2023-09-23T07:09:21.627 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-test-2:18.0.0-6334.gee696c23.el8.x86_64 6/6 2023-09-23T07:09:21.953 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-test-2:18.0.0-6334.gee696c23.el8.x86_64 6/6 2023-09-23T07:09:21.953 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-test-2:18.0.0-6334.gee696c23.el8.x86_64 1/6 2023-09-23T07:09:21.953 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libcephsqlite-2:18.0.0-6334.gee696c23.el8.x86_64 2/6 2023-09-23T07:09:21.953 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-23T07:09:21.953 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-23T07:09:21.954 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-23T07:09:22.631 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-23T07:09:22.631 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:22.631 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:09:22.631 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-test-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:22.631 INFO:teuthology.orchestra.run.smithi140.stdout: jq-1.6-7.el8.x86_64 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout: libcephsqlite-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:22.632 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:22.864 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph 2023-09-23T07:09:23.731 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:00 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:25.304 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: ceph x86_64 2:18.0.0-6334.gee696c23.el8 ceph 6.4 k 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mds x86_64 2:18.0.0-6334.gee696c23.el8 ceph 2.3 M 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr x86_64 2:18.0.0-6334.gee696c23.el8 ceph 1.6 M 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 242 k 2023-09-23T07:09:25.308 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mon x86_64 2:18.0.0-6334.gee696c23.el8 ceph 4.6 M 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-osd x86_64 2:18.0.0-6334.gee696c23.el8 ceph 18 M 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-23T07:09:25.309 INFO:teuthology.orchestra.run.smithi140.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-23T07:09:25.310 INFO:teuthology.orchestra.run.smithi140.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout:Install 32 Packages 2023-09-23T07:09:25.311 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:25.312 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 30 M 2023-09-23T07:09:25.312 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 121 M 2023-09-23T07:09:25.312 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:09:26.156 INFO:teuthology.orchestra.run.smithi140.stdout:(1/32): ceph-mds-18.0.0-6334.gee696c23.el8.x86_ 6.1 MB/s | 2.3 MB 00:00 2023-09-23T07:09:26.188 INFO:teuthology.orchestra.run.smithi140.stdout:(2/32): ceph-18.0.0-6334.gee696c23.el8.x86_64.r 16 kB/s | 6.4 kB 00:00 2023-09-23T07:09:26.230 INFO:teuthology.orchestra.run.smithi140.stdout:(3/32): ceph-mgr-18.0.0-6334.gee696c23.el8.x86_ 3.6 MB/s | 1.6 MB 00:00 2023-09-23T07:09:26.322 INFO:teuthology.orchestra.run.smithi140.stdout:(4/32): ceph-mgr-modules-core-18.0.0-6334.gee69 2.6 MB/s | 242 kB 00:00 2023-09-23T07:09:26.680 INFO:teuthology.orchestra.run.smithi140.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 438 kB/s | 157 kB 00:00 2023-09-23T07:09:26.773 INFO:teuthology.orchestra.run.smithi140.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 424 kB/s | 39 kB 00:00 2023-09-23T07:09:27.023 INFO:teuthology.orchestra.run.smithi140.stdout:(7/32): ceph-osd-18.0.0-6334.gee696c23.el8.x86_ 21 MB/s | 18 MB 00:00 2023-09-23T07:09:27.048 INFO:teuthology.orchestra.run.smithi140.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 376 kB/s | 103 kB 00:00 2023-09-23T07:09:27.273 INFO:teuthology.orchestra.run.smithi140.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 2.0 MB/s | 457 kB 00:00 2023-09-23T07:09:27.340 INFO:teuthology.orchestra.run.smithi140.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 180 kB/s | 57 kB 00:00 2023-09-23T07:09:27.649 INFO:teuthology.orchestra.run.smithi140.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 662 kB/s | 249 kB 00:00 2023-09-23T07:09:27.741 INFO:teuthology.orchestra.run.smithi140.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-09-23T07:09:27.866 INFO:teuthology.orchestra.run.smithi140.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.2 MB/s | 641 kB 00:00 2023-09-23T07:09:28.275 INFO:teuthology.orchestra.run.smithi140.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 82 kB/s | 44 kB 00:00 2023-09-23T07:09:28.550 INFO:teuthology.orchestra.run.smithi140.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 273 kB/s | 185 kB 00:00 2023-09-23T07:09:28.609 INFO:teuthology.orchestra.run.smithi140.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 517 kB/s | 173 kB 00:00 2023-09-23T07:09:28.734 INFO:teuthology.orchestra.run.smithi140.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2023-09-23T07:09:28.826 INFO:teuthology.orchestra.run.smithi140.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2023-09-23T07:09:28.859 INFO:teuthology.orchestra.run.smithi140.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2023-09-23T07:09:28.968 INFO:teuthology.orchestra.run.smithi140.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 344 kB/s | 49 kB 00:00 2023-09-23T07:09:28.985 INFO:teuthology.orchestra.run.smithi140.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2023-09-23T07:09:29.094 INFO:teuthology.orchestra.run.smithi140.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-09-23T07:09:29.127 INFO:teuthology.orchestra.run.smithi140.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-09-23T07:09:29.211 INFO:teuthology.orchestra.run.smithi140.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2023-09-23T07:09:29.244 INFO:teuthology.orchestra.run.smithi140.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 205 kB/s | 24 kB 00:00 2023-09-23T07:09:29.336 INFO:teuthology.orchestra.run.smithi140.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-09-23T07:09:29.370 INFO:teuthology.orchestra.run.smithi140.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 220 kB/s | 27 kB 00:00 2023-09-23T07:09:29.487 INFO:teuthology.orchestra.run.smithi140.stdout:(28/32): ceph-mon-18.0.0-6334.gee696c23.el8.x86 1.4 MB/s | 4.6 MB 00:03 2023-09-23T07:09:29.529 INFO:teuthology.orchestra.run.smithi140.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-09-23T07:09:29.554 INFO:teuthology.orchestra.run.smithi140.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2023-09-23T07:09:29.663 INFO:teuthology.orchestra.run.smithi140.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 171 kB/s | 23 kB 00:00 2023-09-23T07:09:29.996 INFO:teuthology.orchestra.run.smithi140.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 168 kB/s | 85 kB 00:00 2023-09-23T07:09:30.000 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:09:30.001 INFO:teuthology.orchestra.run.smithi140.stdout:Total 6.5 MB/s | 30 MB 00:04 2023-09-23T07:09:30.086 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:09:30.124 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:09:30.124 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:09:30.425 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:09:30.428 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:09:31.656 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:09:31.882 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-23T07:09:32.118 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-23T07:09:32.353 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-23T07:09:32.575 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-23T07:09:32.781 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-23T07:09:33.013 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-23T07:09:33.238 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-23T07:09:33.474 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-23T07:09:33.742 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-23T07:09:33.996 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-23T07:09:34.267 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-23T07:09:34.532 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-23T07:09:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-23T07:09:35.279 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-23T07:09:35.500 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-23T07:09:35.777 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-23T07:09:36.027 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-23T07:09:36.387 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-23T07:09:36.623 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-23T07:09:36.870 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-23T07:09:37.089 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-23T07:09:37.328 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-23T07:09:37.596 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-23T07:09:37.855 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-23T07:09:38.197 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-23T07:09:38.559 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-23T07:09:38.959 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6334.gee696c23.el8. 27/32 2023-09-23T07:09:39.186 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 28/32 2023-09-23T07:09:40.784 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 28/32 2023-09-23T07:09:40.969 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-osd-2:18.0.0-6334.gee696c23.el8.x86_64 29/32 2023-09-23T07:09:41.429 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-osd-2:18.0.0-6334.gee696c23.el8.x86_64 29/32 2023-09-23T07:09:41.625 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mon-2:18.0.0-6334.gee696c23.el8.x86_64 30/32 2023-09-23T07:09:41.851 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mon-2:18.0.0-6334.gee696c23.el8.x86_64 30/32 2023-09-23T07:09:42.000 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mds-2:18.0.0-6334.gee696c23.el8.x86_64 31/32 2023-09-23T07:09:42.018 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mds-2:18.0.0-6334.gee696c23.el8.x86_64 31/32 2023-09-23T07:09:42.166 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-2:18.0.0-6334.gee696c23.el8.x86_64 32/32 2023-09-23T07:09:42.577 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-2:18.0.0-6334.gee696c23.el8.x86_64 32/32 2023-09-23T07:09:42.577 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-2:18.0.0-6334.gee696c23.el8.x86_64 1/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mds-2:18.0.0-6334.gee696c23.el8.x86_64 2/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 3/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mon-2:18.0.0-6334.gee696c23.el8.x86_64 4/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-osd-2:18.0.0-6334.gee696c23.el8.x86_64 5/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6334.gee696c23.el8. 6/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-23T07:09:42.578 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-23T07:09:42.579 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-23T07:09:42.580 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-23T07:09:42.581 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-23T07:09:43.691 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-23T07:09:43.714 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:43.714 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:09:43.714 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:43.714 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mds-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-modules-core-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mon-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-osd-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-23T07:09:43.715 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-portend-2.6-1.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-23T07:09:43.716 INFO:teuthology.orchestra.run.smithi140.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:43.717 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:43.986 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-base 2023-09-23T07:09:44.851 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:21 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:46.240 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:09:46.420 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:09:46.421 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:09:46.422 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:46.522 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install cephadm 2023-09-23T07:09:47.379 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:24 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:48.944 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:09:48.946 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:48.946 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:09:48.946 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:48.946 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout: cephadm noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 183 k 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:09:48.947 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:48.948 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 183 k 2023-09-23T07:09:48.948 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 729 k 2023-09-23T07:09:48.948 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:09:49.245 INFO:teuthology.orchestra.run.smithi140.stdout:cephadm-18.0.0-6334.gee696c23.el8.noarch.rpm 614 kB/s | 183 kB 00:00 2023-09-23T07:09:49.245 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:09:49.245 INFO:teuthology.orchestra.run.smithi140.stdout:Total 611 kB/s | 183 kB 00:00 2023-09-23T07:09:49.246 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:09:49.254 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:09:49.254 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:09:49.256 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:09:49.259 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:09:49.305 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:09:49.903 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: cephadm-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:09:50.038 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : cephadm-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:09:50.264 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: cephadm-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : cephadm-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout: cephadm-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:51.245 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:53.195 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-immutable-object-cache 2023-09-23T07:09:54.072 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:31 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:55.647 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:09:55.647 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repo Size 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6334.gee696c23.el8 ceph 167 k 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:09:55.648 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:55.649 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 167 k 2023-09-23T07:09:55.649 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 512 k 2023-09-23T07:09:55.649 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:09:55.976 INFO:teuthology.orchestra.run.smithi140.stdout:ceph-immutable-object-cache-18.0.0-6334.gee696c 511 kB/s | 167 kB 00:00 2023-09-23T07:09:55.976 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:09:55.976 INFO:teuthology.orchestra.run.smithi140.stdout:Total 509 kB/s | 167 kB 00:00 2023-09-23T07:09:55.977 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:09:55.988 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:09:55.988 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:09:56.034 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:09:56.037 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:09:56.143 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:09:56.306 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6334.gee696c23. 1/1 2023-09-23T07:09:56.732 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6334.gee696c23. 1/1 2023-09-23T07:09:57.404 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6334.gee696c23. 1/1 2023-09-23T07:09:57.404 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:57.404 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:09:57.404 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-immutable-object-cache-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:09:57.404 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:09:57.405 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:09:57.638 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-mgr 2023-09-23T07:09:58.499 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:35 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:09:59.889 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:10:00.071 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:00.072 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:10:00.072 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:00.179 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-mgr-dashboard 2023-09-23T07:10:01.035 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:38 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:02.604 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:02.605 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:02.605 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:02.605 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 1.7 M 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 23 k 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 15 k 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:02.606 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:02.607 INFO:teuthology.orchestra.run.smithi140.stdout:Install 6 Packages 2023-09-23T07:10:02.607 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:02.607 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 2.1 M 2023-09-23T07:10:02.607 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 8.5 M 2023-09-23T07:10:02.608 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:03.470 INFO:teuthology.orchestra.run.smithi140.stdout:(1/6): ceph-grafana-dashboards-18.0.0-6334.gee6 140 kB/s | 23 kB 00:00 2023-09-23T07:10:03.634 INFO:teuthology.orchestra.run.smithi140.stdout:(2/6): ceph-prometheus-alerts-18.0.0-6334.gee69 44 kB/s | 15 kB 00:00 2023-09-23T07:10:03.679 INFO:teuthology.orchestra.run.smithi140.stdout:(3/6): ceph-mgr-dashboard-18.0.0-6334.gee696c23 4.7 MB/s | 1.7 MB 00:00 2023-09-23T07:10:03.704 INFO:teuthology.orchestra.run.smithi140.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2023-09-23T07:10:03.963 INFO:teuthology.orchestra.run.smithi140.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 102 kB/s | 34 kB 00:00 2023-09-23T07:10:04.163 INFO:teuthology.orchestra.run.smithi140.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 406 kB/s | 196 kB 00:00 2023-09-23T07:10:04.164 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:04.165 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.3 MB/s | 2.1 MB 00:01 2023-09-23T07:10:04.176 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:04.186 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:04.186 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:04.273 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:04.277 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:04.577 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:04.760 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-23T07:10:04.997 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-23T07:10:05.190 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-23T07:10:05.344 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6334.gee696c23.el8.n 4/6 2023-09-23T07:10:05.706 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6334.gee696c23.el8. 5/6 2023-09-23T07:10:05.906 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6334.gee696c23.el8.noarc 6/6 2023-09-23T07:10:06.213 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6334.gee696c23.el8.noarc 6/6 2023-09-23T07:10:06.214 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6334.gee696c23.el8. 1/6 2023-09-23T07:10:06.214 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6334.gee696c23.el8.noarc 2/6 2023-09-23T07:10:06.214 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6334.gee696c23.el8.n 3/6 2023-09-23T07:10:06.214 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-23T07:10:06.214 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-grafana-dashboards-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-dashboard-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-prometheus-alerts-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-23T07:10:06.847 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:06.848 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:07.000 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-23T07:10:07.860 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:01:44 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:09.443 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:09.444 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================================ 2023-09-23T07:10:09.444 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================================ 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 7.4 M 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-23T07:10:09.445 INFO:teuthology.orchestra.run.smithi140.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================================ 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout:Install 8 Packages 2023-09-23T07:10:09.446 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:09.447 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 35 M 2023-09-23T07:10:09.447 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 210 M 2023-09-23T07:10:09.447 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:10.853 INFO:teuthology.orchestra.run.smithi140.stdout:(1/8): ceph-mgr-diskprediction-local-18.0.0-633 13 MB/s | 7.4 MB 00:00 2023-09-23T07:10:11.128 INFO:teuthology.orchestra.run.smithi140.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.9 MB 00:00 2023-09-23T07:10:11.195 INFO:teuthology.orchestra.run.smithi140.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.3 MB/s | 4.7 MB 00:00 2023-09-23T07:10:11.220 INFO:teuthology.orchestra.run.smithi140.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-09-23T07:10:11.513 INFO:teuthology.orchestra.run.smithi140.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.7 MB/s | 3.7 MB 00:00 2023-09-23T07:10:11.754 INFO:teuthology.orchestra.run.smithi140.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2023-09-23T07:10:11.821 INFO:teuthology.orchestra.run.smithi140.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 559 kB/s | 172 kB 00:00 2023-09-23T07:10:12.042 INFO:teuthology.orchestra.run.smithi140.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 16 MB/s | 14 MB 00:00 2023-09-23T07:10:12.043 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:12.043 INFO:teuthology.orchestra.run.smithi140.stdout:Total 14 MB/s | 35 MB 00:02 2023-09-23T07:10:12.275 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:12.292 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:12.293 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:12.685 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:12.688 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:13.270 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:13.453 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-23T07:10:13.590 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-23T07:10:14.728 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-23T07:10:15.335 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-23T07:10:15.527 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-23T07:10:16.141 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-23T07:10:16.327 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-23T07:10:17.015 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-23T07:10:17.373 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-23T07:10:17.682 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-23T07:10:19.459 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-23T07:10:21.075 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-23T07:10:21.365 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6334.gee696c2 8/8 2023-09-23T07:10:21.668 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6334.gee696c2 8/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6334.gee696c2 1/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-23T07:10:21.669 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-23T07:10:22.321 INFO:teuthology.orchestra.run.smithi140.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-23T07:10:22.322 INFO:teuthology.orchestra.run.smithi140.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-23T07:10:22.322 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:22.322 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:22.450 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-mgr-rook 2023-09-23T07:10:23.305 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:00 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:24.874 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-rook noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 47 k 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:10:24.876 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-23T07:10:24.877 INFO:teuthology.orchestra.run.smithi140.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout:Install 13 Packages 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 1.9 M 2023-09-23T07:10:24.878 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 25 M 2023-09-23T07:10:24.879 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:25.557 INFO:teuthology.orchestra.run.smithi140.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 84 kB/s | 20 kB 00:00 2023-09-23T07:10:25.587 INFO:teuthology.orchestra.run.smithi140.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 99 kB/s | 27 kB 00:00 2023-09-23T07:10:25.730 INFO:teuthology.orchestra.run.smithi140.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 732 kB/s | 126 kB 00:00 2023-09-23T07:10:25.790 INFO:teuthology.orchestra.run.smithi140.stdout:(4/13): ceph-mgr-rook-18.0.0-6334.gee696c23.el8 101 kB/s | 47 kB 00:00 2023-09-23T07:10:25.822 INFO:teuthology.orchestra.run.smithi140.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 470 kB/s | 110 kB 00:00 2023-09-23T07:10:26.039 INFO:teuthology.orchestra.run.smithi140.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 171 kB/s | 43 kB 00:00 2023-09-23T07:10:26.064 INFO:teuthology.orchestra.run.smithi140.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 464 kB/s | 155 kB 00:00 2023-09-23T07:10:26.214 INFO:teuthology.orchestra.run.smithi140.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 84 kB/s | 33 kB 00:00 2023-09-23T07:10:26.315 INFO:teuthology.orchestra.run.smithi140.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 59 kB/s | 16 kB 00:00 2023-09-23T07:10:26.449 INFO:teuthology.orchestra.run.smithi140.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 211 kB/s | 81 kB 00:00 2023-09-23T07:10:26.515 INFO:teuthology.orchestra.run.smithi140.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 321 kB/s | 64 kB 00:00 2023-09-23T07:10:26.557 INFO:teuthology.orchestra.run.smithi140.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 566 kB/s | 61 kB 00:00 2023-09-23T07:10:26.624 INFO:teuthology.orchestra.run.smithi140.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.9 MB/s | 1.2 MB 00:00 2023-09-23T07:10:26.627 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:26.627 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-09-23T07:10:26.686 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:26.702 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:26.703 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:26.790 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:26.793 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:27.358 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:27.645 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-23T07:10:27.834 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-23T07:10:28.069 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-23T07:10:28.374 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-23T07:10:28.614 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-23T07:10:28.904 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-23T07:10:29.212 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-23T07:10:29.462 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-23T07:10:30.518 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-23T07:10:30.823 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-23T07:10:31.065 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-23T07:10:31.309 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-23T07:10:31.508 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-rook-2:18.0.0-6334.gee696c23.el8.noarch 13/13 2023-09-23T07:10:31.804 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6334.gee696c23.el8.noarch 13/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6334.gee696c23.el8.noarch 1/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-23T07:10:31.805 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-23T07:10:31.806 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-23T07:10:32.541 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-rook-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-23T07:10:32.542 INFO:teuthology.orchestra.run.smithi140.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:32.543 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:32.711 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-mgr-cephadm 2023-09-23T07:10:33.570 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:10 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:35.157 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:35.158 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:35.158 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:35.158 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:35.158 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 132 k 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout:Install 5 Packages 2023-09-23T07:10:35.159 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:35.160 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 6.0 M 2023-09-23T07:10:35.160 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 26 M 2023-09-23T07:10:35.160 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:35.674 INFO:teuthology.orchestra.run.smithi140.stdout:(1/5): ceph-mgr-cephadm-18.0.0-6334.gee696c23.e 688 kB/s | 132 kB 00:00 2023-09-23T07:10:35.790 INFO:teuthology.orchestra.run.smithi140.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2023-09-23T07:10:35.966 INFO:teuthology.orchestra.run.smithi140.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 9.9 MB/s | 4.8 MB 00:00 2023-09-23T07:10:35.991 INFO:teuthology.orchestra.run.smithi140.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 301 kB/s | 60 kB 00:00 2023-09-23T07:10:36.033 INFO:teuthology.orchestra.run.smithi140.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2023-09-23T07:10:36.034 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:36.034 INFO:teuthology.orchestra.run.smithi140.stdout:Total 6.8 MB/s | 6.0 MB 00:00 2023-09-23T07:10:36.089 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:36.100 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:36.100 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:36.204 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:36.208 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:36.562 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:36.906 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-23T07:10:37.887 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-23T07:10:38.295 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-23T07:10:38.588 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-23T07:10:38.807 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6334.gee696c23.el8.noarch 5/5 2023-09-23T07:10:39.096 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6334.gee696c23.el8.noarch 5/5 2023-09-23T07:10:39.096 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6334.gee696c23.el8.noarch 1/5 2023-09-23T07:10:39.097 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-23T07:10:39.097 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-23T07:10:39.097 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-cephadm-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-23T07:10:39.750 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:39.751 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:39.873 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-fuse 2023-09-23T07:10:40.731 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:17 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:42.310 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:42.310 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-fuse x86_64 2:18.0.0-6334.gee696c23.el8 ceph 912 k 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout:Install 2 Packages 2023-09-23T07:10:42.311 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:42.312 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 995 k 2023-09-23T07:10:42.312 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 3.1 M 2023-09-23T07:10:42.312 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:42.566 INFO:teuthology.orchestra.run.smithi140.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 430 kB/s | 83 kB 00:00 2023-09-23T07:10:42.907 INFO:teuthology.orchestra.run.smithi140.stdout:(2/2): ceph-fuse-18.0.0-6334.gee696c23.el8.x86_ 1.7 MB/s | 912 kB 00:00 2023-09-23T07:10:42.908 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:42.909 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.6 MB/s | 995 kB 00:00 2023-09-23T07:10:42.919 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:42.941 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:42.941 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:43.027 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:43.030 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:43.232 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:43.543 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-23T07:10:43.789 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 2/2 2023-09-23T07:10:44.220 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 2/2 2023-09-23T07:10:44.220 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 1/2 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:44.900 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:45.127 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-volume 2023-09-23T07:10:45.997 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:22 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:47.578 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:47.579 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-volume noarch 2:18.0.0-6334.gee696c23.el8 ceph-noarch 261 k 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:47.580 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 261 k 2023-09-23T07:10:47.581 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 1.3 M 2023-09-23T07:10:47.581 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:47.795 INFO:teuthology.orchestra.run.smithi140.stdout:ceph-volume-18.0.0-6334.gee696c23.el8.noarch.rp 1.2 MB/s | 261 kB 00:00 2023-09-23T07:10:47.795 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:47.796 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.2 MB/s | 261 kB 00:00 2023-09-23T07:10:47.796 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:47.806 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:47.806 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:47.837 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:47.840 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:48.020 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:48.289 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ceph-volume-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:10:48.724 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ceph-volume-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:10:49.302 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ceph-volume-2:18.0.0-6334.gee696c23.el8.noarch 1/1 2023-09-23T07:10:49.303 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:49.303 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:49.303 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-volume-2:18.0.0-6334.gee696c23.el8.noarch 2023-09-23T07:10:49.303 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:49.303 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:49.537 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install librados-devel 2023-09-23T07:10:50.395 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:27 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:51.974 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:51.974 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repo Size 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout: librados-devel x86_64 2:18.0.0-6334.gee696c23.el8 ceph 139 k 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:51.975 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:51.976 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:10:51.976 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:51.976 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 139 k 2023-09-23T07:10:51.977 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 513 k 2023-09-23T07:10:51.977 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:52.788 INFO:teuthology.orchestra.run.smithi140.stdout:librados-devel-18.0.0-6334.gee696c23.el8.x86_64 171 kB/s | 139 kB 00:00 2023-09-23T07:10:52.789 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:52.789 INFO:teuthology.orchestra.run.smithi140.stdout:Total 171 kB/s | 139 kB 00:00 2023-09-23T07:10:52.790 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:52.796 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:52.796 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:52.852 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:52.855 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:10:53.092 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:10:53.362 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : librados-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:10:53.669 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: librados-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:10:54.253 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : librados-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:10:54.253 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:54.254 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:10:54.254 INFO:teuthology.orchestra.run.smithi140.stdout: librados-devel-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:10:54.254 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:54.254 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:54.458 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install libcephfs2 2023-09-23T07:10:55.323 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:32 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:56.730 INFO:teuthology.orchestra.run.smithi140.stdout:Package libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:10:56.911 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:56.913 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:10:56.913 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:10:57.014 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install libcephfs-devel 2023-09-23T07:10:57.873 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:34 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:10:59.437 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:10:59.438 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:59.438 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repo Size 2023-09-23T07:10:59.438 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout: libcephfs-devel x86_64 2:18.0.0-6334.gee696c23.el8 ceph 31 k 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 31 k 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 139 k 2023-09-23T07:10:59.439 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:10:59.931 INFO:teuthology.orchestra.run.smithi140.stdout:libcephfs-devel-18.0.0-6334.gee696c23.el8.x86_6 63 kB/s | 31 kB 00:00 2023-09-23T07:10:59.931 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:10:59.931 INFO:teuthology.orchestra.run.smithi140.stdout:Total 63 kB/s | 31 kB 00:00 2023-09-23T07:10:59.932 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:10:59.935 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:10:59.935 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:10:59.963 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:10:59.966 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:11:00.023 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:11:00.253 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libcephfs-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:00.542 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libcephfs-devel-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout: libcephfs-devel-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:01.120 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:01.232 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install librados2 2023-09-23T07:11:02.083 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:39 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:03.478 INFO:teuthology.orchestra.run.smithi140.stdout:Package librados2-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:03.663 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:03.665 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:03.665 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:03.777 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install librbd1 2023-09-23T07:11:04.635 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:41 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:06.030 INFO:teuthology.orchestra.run.smithi140.stdout:Package librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:06.218 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:06.220 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:06.220 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:06.323 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install python3-rados 2023-09-23T07:11:07.180 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:44 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:08.566 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rados-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:08.752 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:08.753 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:08.753 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:08.849 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install python3-rgw 2023-09-23T07:11:09.707 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:46 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:11.116 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rgw-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:11.300 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:11.302 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:11.302 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:11.406 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install python3-cephfs 2023-09-23T07:11:12.270 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:49 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:13.662 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-cephfs-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:13.845 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:13.846 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:13.847 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:13.943 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install python3-rbd 2023-09-23T07:11:14.800 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:51 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:16.197 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rbd-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:11:16.378 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:16.379 INFO:teuthology.orchestra.run.smithi140.stdout:Nothing to do. 2023-09-23T07:11:16.380 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:16.492 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install rbd-fuse 2023-09-23T07:11:17.344 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:54 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:18.938 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout: Package Architecture Version Repository Size 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-fuse x86_64 2:18.0.0-6334.gee696c23.el8 ceph 94 k 2023-09-23T07:11:18.939 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 94 k 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 276 k 2023-09-23T07:11:18.940 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:11:19.094 INFO:teuthology.orchestra.run.smithi140.stdout:rbd-fuse-18.0.0-6334.gee696c23.el8.x86_64.rpm 608 kB/s | 94 kB 00:00 2023-09-23T07:11:19.095 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:11:19.095 INFO:teuthology.orchestra.run.smithi140.stdout:Total 603 kB/s | 94 kB 00:00 2023-09-23T07:11:19.095 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:11:19.106 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:11:19.106 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:11:19.154 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:11:19.157 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:11:19.287 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:11:19.541 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : rbd-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:19.892 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : rbd-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:20.552 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:20.775 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install rbd-mirror 2023-09-23T07:11:21.637 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:02:58 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:23.207 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:23.207 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repo Size 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-mirror x86_64 2:18.0.0-6334.gee696c23.el8 ceph 3.6 M 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:11:23.208 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:23.209 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 3.6 M 2023-09-23T07:11:23.209 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 17 M 2023-09-23T07:11:23.209 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:11:26.164 INFO:teuthology.orchestra.run.smithi140.stdout:rbd-mirror-18.0.0-6334.gee696c23.el8.x86_64.rpm 1.2 MB/s | 3.6 MB 00:02 2023-09-23T07:11:26.165 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:11:26.165 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.2 MB/s | 3.6 MB 00:02 2023-09-23T07:11:26.166 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:11:26.179 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:11:26.179 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:11:26.246 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:11:26.250 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:11:26.737 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:11:26.983 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : rbd-mirror-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:27.400 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : rbd-mirror-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-mirror-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:28.071 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:28.294 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install rbd-nbd 2023-09-23T07:11:29.157 INFO:teuthology.orchestra.run.smithi140.stdout:Last metadata expiration check: 0:03:06 ago on Sat 23 Sep 2023 07:08:23 AM UTC. 2023-09-23T07:11:30.738 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout: Package Architecture Version Repository Size 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-nbd x86_64 2:18.0.0-6334.gee696c23.el8 ceph 179 k 2023-09-23T07:11:30.739 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:Install 1 Package 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 179 k 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 564 k 2023-09-23T07:11:30.740 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:11:30.967 INFO:teuthology.orchestra.run.smithi140.stdout:rbd-nbd-18.0.0-6334.gee696c23.el8.x86_64.rpm 789 kB/s | 179 kB 00:00 2023-09-23T07:11:30.967 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:11:30.967 INFO:teuthology.orchestra.run.smithi140.stdout:Total 785 kB/s | 179 kB 00:00 2023-09-23T07:11:30.968 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:11:30.979 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:11:30.979 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:11:31.030 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:11:31.033 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:11:31.146 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:11:31.364 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : rbd-nbd-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:31.639 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : rbd-nbd-2:18.0.0-6334.gee696c23.el8.x86_64 1/1 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout: rbd-nbd-2:18.0.0-6334.gee696c23.el8.x86_64 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:32.260 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:11:32.512 DEBUG:teuthology.parallel:result is None 2023-09-23T07:11:32.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:11:32.902 DEBUG:teuthology.orchestra.run.smithi140:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-23T07:11:32.925 INFO:teuthology.orchestra.run.smithi140.stdout:18.0.0-6334.gee696c23.el8 2023-09-23T07:11:32.926 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6334.gee696c23.el8 2023-09-23T07:11:32.926 INFO:teuthology.task.install:The correct ceph version 18.0.0-6334.gee696c23 is installed. 2023-09-23T07:11:32.926 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-23T07:11:32.926 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:32.927 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-23T07:11:32.994 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-23T07:11:32.995 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:32.995 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/daemon-helper 2023-09-23T07:11:33.062 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-23T07:11:33.129 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-23T07:11:33.129 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:33.129 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-23T07:11:33.196 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-23T07:11:33.263 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-23T07:11:33.263 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:33.263 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/stdin-killer 2023-09-23T07:11:33.330 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-23T07:11:33.397 INFO:teuthology.run_tasks:Running task ceph... 2023-09-23T07:11:33.460 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-23T07:11:33.460 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 777 /var/log/ceph 2023-09-23T07:11:33.487 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-23T07:11:33.487 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-23T07:11:33.553 INFO:tasks.ceph:Creating extra log directories... 2023-09-23T07:11:33.553 DEBUG:teuthology.orchestra.run.smithi140:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-23T07:11:33.628 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-23T07:11:33.628 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-23T07:11:33.629 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649', 'branch': 'wip-62891', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7401649', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-62891'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-62891', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2241, 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-62891', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'targets': {'smithi140.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM7EfadN9AifUj3gzNG/vZ54NYmDnMRc27pZKNKLHzZBLjbo6PApZOFYWXCd3JzsDiy0T0vb/YQkmWSvAIr63Vw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2', 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile', 'sudo ceph osd pool set datapool allow_ec_overwrites true', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-23_06:37:41', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-23T07:11:33.629 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-23T07:11:33.679 DEBUG:teuthology.orchestra.run.smithi140:> sudo install -d -m0777 -- /var/run/ceph 2023-09-23T07:11:33.746 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:33.746 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2023-09-23T07:11:33.805 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-23T07:11:33.805 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_1 2023-09-23T07:11:33.862 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-23T07:11:33.862 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T07:11:33.862 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 6h/6d Inode: 389620 Links: 1 2023-09-23T07:11:33.862 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T07:11:33.863 INFO:teuthology.orchestra.run.smithi140.stdout:Context: system_u:object_r:device_t:s0 2023-09-23T07:11:33.863 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-09-23 07:11:27.150410792 +0000 2023-09-23T07:11:33.863 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-09-23 07:07:10.968527862 +0000 2023-09-23T07:11:33.863 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-09-23 07:07:10.968527862 +0000 2023-09-23T07:11:33.863 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-09-23T07:11:33.863 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-23T07:11:33.929 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-09-23T07:11:33.929 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-09-23T07:11:33.929 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000230864 s, 2.2 MB/s 2023-09-23T07:11:33.930 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-23T07:11:33.990 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_2 2023-09-23T07:11:34.048 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-23T07:11:34.048 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 6h/6d Inode: 391604 Links: 1 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Context: system_u:object_r:device_t:s0 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-09-23 07:11:27.151410771 +0000 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-09-23 07:07:11.466517300 +0000 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-09-23 07:07:11.466517300 +0000 2023-09-23T07:11:34.049 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-09-23T07:11:34.049 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-23T07:11:34.113 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-09-23T07:11:34.113 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-09-23T07:11:34.113 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000237492 s, 2.2 MB/s 2023-09-23T07:11:34.114 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-23T07:11:34.174 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_3 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 6h/6d Inode: 395460 Links: 1 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Context: system_u:object_r:device_t:s0 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-09-23 07:11:27.151410771 +0000 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-09-23 07:07:12.008505805 +0000 2023-09-23T07:11:34.232 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-09-23 07:07:12.008505805 +0000 2023-09-23T07:11:34.233 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-09-23T07:11:34.233 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-23T07:11:34.298 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-09-23T07:11:34.298 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-09-23T07:11:34.298 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000190815 s, 2.7 MB/s 2023-09-23T07:11:34.299 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-23T07:11:34.358 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_4 2023-09-23T07:11:34.416 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-23T07:11:34.416 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T07:11:34.417 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 6h/6d Inode: 394167 Links: 1 2023-09-23T07:11:34.417 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T07:11:34.417 INFO:teuthology.orchestra.run.smithi140.stdout:Context: system_u:object_r:device_t:s0 2023-09-23T07:11:34.417 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-09-23 07:11:27.151410771 +0000 2023-09-23T07:11:34.418 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-09-23 07:07:12.512495116 +0000 2023-09-23T07:11:34.418 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-09-23 07:07:12.512495116 +0000 2023-09-23T07:11:34.418 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-09-23T07:11:34.418 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-23T07:11:34.481 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-09-23T07:11:34.482 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-09-23T07:11:34.482 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000219628 s, 2.3 MB/s 2023-09-23T07:11:34.483 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-23T07:11:34.541 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T07:11:34.542 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi140.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-23T07:11:34.542 INFO:tasks.ceph:Generating config... 2023-09-23T07:11:34.543 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-09-23T07:11:34.543 INFO:tasks.ceph:[client] rbd default features = 61 2023-09-23T07:11:34.543 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-23T07:11:34.543 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-09-23T07:11:34.543 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-23T07:11:34.543 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-23T07:11:34.543 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-23T07:11:34.543 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-23T07:11:34.544 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-23T07:11:34.544 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] enable experimental unrecoverable data corrupting features = * 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] osd debug randomize hobject sort order = False 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-23T07:11:34.545 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-23T07:11:34.545 INFO:tasks.ceph:Setting up mon.a... 2023-09-23T07:11:34.546 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-23T07:11:34.634 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/ceph.keyring 2023-09-23T07:11:34.636 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-23T07:11:34.681 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-23T07:11:34.708 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.140')] 2023-09-23T07:11:34.709 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.140', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2023-09-23T07:11:34.709 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:34.709 DEBUG:teuthology.orchestra.run.smithi140:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-23T07:11:34.766 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.140 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:34.849 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:34.849 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: generated fsid f962b619-b16f-4c4b-b0ad-bbfa0665eb75 2023-09-23T07:11:34.849 INFO:teuthology.orchestra.run.smithi140.stdout:setting min_mon_release = pacific 2023-09-23T07:11:34.849 INFO:teuthology.orchestra.run.smithi140.stdout:epoch 0 2023-09-23T07:11:34.849 INFO:teuthology.orchestra.run.smithi140.stdout:fsid f962b619-b16f-4c4b-b0ad-bbfa0665eb75 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:last_changed 2023-09-23T07:11:34.849433+0000 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:created 2023-09-23T07:11:34.849433+0000 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:min_mon_release 16 (pacific) 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:election_strategy: 1 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:0: [v2:172.21.15.140:3300/0,v1:172.21.15.140:6789/0] mon.a 2023-09-23T07:11:34.850 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-09-23T07:11:34.851 DEBUG:teuthology.orchestra.run.smithi140:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-23T07:11:34.909 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f962b619-b16f-4c4b-b0ad-bbfa0665eb75... 2023-09-23T07:11:34.909 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout:[global] 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: chdir = "" 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: auth supported = cephx 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: filestore xattr use omap = true 2023-09-23T07:11:34.995 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: mon clock drift allowed = 1.000 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: osd crush chooseleaf type = 0 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: auth debug = true 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on old message = true 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on bug = true 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: mon pg warn max object skew = 0 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.996 INFO:teuthology.orchestra.run.smithi140.stdout: # disable pg_autoscaler by default for new pools 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default size = 2 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow primary affinity = true 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow pg remap = true 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on legacy crush tunables = false 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on crush straw calc version zero = false 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on no sortbitwise = false 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on osd down out interval zero = false 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on too few osds = false 2023-09-23T07:11:34.997 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: mon_allow_pool_size_one = true 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: osd default data pool replay window = 5 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: mon allow pool delete = true 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: mon cluster log file level = debug 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: debug asserts on shutdown = true 2023-09-23T07:11:34.998 INFO:teuthology.orchestra.run.smithi140.stdout: mon health detail to clog = false 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: mon host = 172.21.15.140 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: mon client directed command retry = 5 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on pool no app = False 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: ms inject socket failures = 5000 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: fsid = f962b619-b16f-4c4b-b0ad-bbfa0665eb75 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout:[osd] 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: osd journal size = 100 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub load threshold = 5.0 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub max interval = 600 2023-09-23T07:11:34.999 INFO:teuthology.orchestra.run.smithi140.stdout: osd mclock profile = high_recovery_ops 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd recover clone overlap = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd recovery max chunk = 1048576 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug shutdown = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug op order = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug verify stray on activate = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd open classes on start = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug pg log writeout = true 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: osd deep scrub update digest min age = 30 2023-09-23T07:11:35.000 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: osd map max advance = 10 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: journal zero on create = true 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: filestore ondisk finisher threads = 3 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: filestore apply finisher threads = 3 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: bdev debug aio = true 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug misdirected ops = true 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore block size = 96636764160 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore compression algorithm = zstd 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore compression mode = aggressive 2023-09-23T07:11:35.001 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore fsck on mount = True 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluefs = 1/20 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluestore = 1/20 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: debug osd = 20 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: debug rocksdb = 4/10 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: enable experimental unrecoverable data corrupting features = * 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd backfillfull_ratio = 0.85 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd full ratio = 0.9 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd nearfull ratio = 0.8 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug randomize hobject sort order = False 2023-09-23T07:11:35.002 INFO:teuthology.orchestra.run.smithi140.stdout: osd failsafe full ratio = 0.95 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: osd objectstore = bluestore 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: osd shutdown pgref assert = True 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout:[mgr] 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: debug mgr = 20 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: mgr/telemetry/nag = false 2023-09-23T07:11:35.003 INFO:teuthology.orchestra.run.smithi140.stdout: debug rbd = 20 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout:[mon] 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: debug paxos = 20 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon data avail warn = 5 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon mgr mkfs grace = 240 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd reporter subtree level = osd 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd prime pg temp = true 2023-09-23T07:11:35.004 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: auth mon ticket ttl = 660 # 11m 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: auth service ticket ttl = 240 # 4m 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: # don't complain about insecure global_id in the test suite 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: # 1m isn't quite enough 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: mon_down_mkfs_grace = 2m 2023-09-23T07:11:35.005 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_filestore_osds = false 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout:[client] 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: rgw cache enabled = true 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable ops log = true 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable usage log = true 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: rbd default data pool = datapool 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout: rbd default features = 61 2023-09-23T07:11:35.006 INFO:teuthology.orchestra.run.smithi140.stdout:[mon.a] 2023-09-23T07:11:35.007 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-23T07:11:35.007 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-23T07:11:35.105 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-23T07:11:35.105 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:35.105 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-23T07:11:35.123 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:35.123 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-23T07:11:35.181 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T07:11:35.181 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:35.181 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-23T07:11:35.181 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-23T07:11:35.257 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:35.258 DEBUG:teuthology.orchestra.run.smithi140:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:35.316 INFO:tasks.ceph:Setting up mon nodes... 2023-09-23T07:11:35.316 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-23T07:11:35.316 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-23T07:11:35.410 INFO:teuthology.orchestra.run.smithi140.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-23T07:11:35.412 INFO:tasks.ceph:Setting up mds nodes... 2023-09-23T07:11:35.413 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-23T07:11:35.413 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-23T07:11:35.457 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-23T07:11:35.469 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-23T07:11:35.469 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi140.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-23T07:11:35.470 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-23T07:11:35.496 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T07:11:35.496 INFO:tasks.ceph:role: osd.0 2023-09-23T07:11:35.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T07:11:35.496 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T07:11:35.571 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-09-23T07:11:35.572 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T07:11:35.572 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T07:11:35.572 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T07:11:35.572 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T07:11:36.002 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-09-23T07:11:36.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-09-23T07:11:36.004 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-23T07:11:36.179 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-23T07:11:36.213 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-23T07:11:36.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T07:11:36.281 INFO:tasks.ceph:role: osd.1 2023-09-23T07:11:36.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T07:11:36.281 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-23T07:11:36.356 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T07:11:36.357 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T07:11:36.776 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-09-23T07:11:36.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-09-23T07:11:36.778 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-23T07:11:36.888 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-23T07:11:36.920 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-23T07:11:36.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T07:11:36.989 INFO:tasks.ceph:role: osd.2 2023-09-23T07:11:36.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T07:11:36.989 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T07:11:37.064 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T07:11:37.065 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T07:11:37.065 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T07:11:37.507 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-09-23T07:11:37.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-09-23T07:11:37.509 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-23T07:11:37.671 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-23T07:11:37.703 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:37.806 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.806+0000 7f0d4d169640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:37.806 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.806+0000 7f0d4d169640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:37.807 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.807+0000 7f0d4d169640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:37.807 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.807+0000 7f0d4d169640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-23T07:11:37.809 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.808+0000 7f0d4d169640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-23T07:11:37.809 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.808+0000 7f0d4d169640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T07:11:37.809 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.808+0000 7f0d4d169640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T07:11:37.809 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.808+0000 7f0d4d169640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T07:11:37.814 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:37.813+0000 7f0d4d169640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-23T07:11:40.669 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-23T07:11:40.703 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:40.795 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.794+0000 7fec8a985640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:40.795 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.794+0000 7fec8a985640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:40.795 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.795+0000 7fec8a985640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:40.796 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.796+0000 7fec8a985640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-23T07:11:40.796 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.796+0000 7fec8a985640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-23T07:11:40.796 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.796+0000 7fec8a985640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T07:11:40.797 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.796+0000 7fec8a985640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T07:11:40.797 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.796+0000 7fec8a985640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T07:11:40.797 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:40.797+0000 7fec8a985640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-23T07:11:43.645 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-23T07:11:43.678 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:43.770 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.770+0000 7ff89c198640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:43.771 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.770+0000 7ff89c198640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:43.771 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.771+0000 7ff89c198640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:43.773 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.771+0000 7ff89c198640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-23T07:11:43.773 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.772+0000 7ff89c198640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-23T07:11:43.773 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.772+0000 7ff89c198640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T07:11:43.773 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.772+0000 7ff89c198640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T07:11:43.774 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.772+0000 7ff89c198640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T07:11:43.774 INFO:teuthology.orchestra.run.smithi140.stderr:2023-09-23T07:11:43.772+0000 7ff89c198640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-23T07:11:46.620 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-23T07:11:46.652 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-23T07:11:46.653 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:46.653 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-23T07:11:46.723 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:46.723 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-23T07:11:46.793 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:46.793 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-23T07:11:46.862 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:46.862 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-23T07:11:46.930 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:46.930 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-23T07:11:46.988 INFO:tasks.ceph:Adding keys to all mons... 2023-09-23T07:11:46.988 DEBUG:teuthology.orchestra.run.smithi140:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-23T07:11:47.054 INFO:teuthology.orchestra.run.smithi140.stdout:[mgr.x] 2023-09-23T07:11:47.054 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCnjw5lKoR4GBAAzKEJ37I/WCmgRNYXWiXBig== 2023-09-23T07:11:47.054 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.0] 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCpjw5lVIQqMBAAl/njw8wIPN1EeK0ab9Pc/Q== 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.1] 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCsjw5lxj58LxAAsB7uJ6p9m8fFpTvATfWNYg== 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.2] 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCvjw5lyUMILhAADhS7kN8uaQmnKOynl8xSUA== 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout:[client.0] 2023-09-23T07:11:47.055 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCnjw5l4FxFGxAAVh3A2UfcFSSyzJ/OrgkOAg== 2023-09-23T07:11:47.056 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-23T07:11:47.151 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T07:11:47.204 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T07:11:47.258 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T07:11:47.312 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-23T07:11:47.368 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-23T07:11:47.369 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-23T07:11:47.395 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-23T07:11:47.494 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-23T07:11:47.521 DEBUG:teuthology.orchestra.run.smithi140:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-23T07:11:47.579 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-23T07:11:47.579 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-23T07:11:47.579 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-23T07:11:47.621 INFO:tasks.ceph.mon.a:Started 2023-09-23T07:11:47.622 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-23T07:11:47.622 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-23T07:11:47.622 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-23T07:11:47.624 INFO:tasks.ceph.mgr.x:Started 2023-09-23T07:11:47.624 INFO:tasks.ceph:Setting crush tunables to default 2023-09-23T07:11:47.624 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd crush tunables default 2023-09-23T07:11:47.847 INFO:teuthology.orchestra.run.smithi140.stderr:adjusted tunables profile to default 2023-09-23T07:11:47.859 INFO:tasks.ceph:check_enable_crimson: False 2023-09-23T07:11:47.859 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-23T07:11:47.859 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:47.859 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-23T07:11:47.886 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:47.886 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-23T07:11:47.953 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:11:47.953 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-23T07:11:48.020 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd new bc742385-5f88-44ba-9305-a3dbe250ea5f 0 2023-09-23T07:11:48.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:48.032+0000 7f55dee71200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-23T07:11:48.220 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:48.218+0000 7f55dee71200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-23T07:11:48.276 INFO:teuthology.orchestra.run.smithi140.stdout:0 2023-09-23T07:11:48.289 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd new 3ee30df5-51cc-47c4-b957-eba695b78d36 1 2023-09-23T07:11:48.501 INFO:teuthology.orchestra.run.smithi140.stdout:1 2023-09-23T07:11:48.513 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd new da2f6412-e3fc-497b-8e52-4d4661cf03c6 2 2023-09-23T07:11:48.595 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:48.594+0000 7f55dee71200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-23T07:11:48.726 INFO:teuthology.orchestra.run.smithi140.stdout:2 2023-09-23T07:11:48.738 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-23T07:11:48.739 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-23T07:11:48.741 INFO:tasks.ceph.osd.0:Started 2023-09-23T07:11:48.741 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-23T07:11:48.741 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-23T07:11:48.743 INFO:tasks.ceph.osd.1:Started 2023-09-23T07:11:48.743 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-23T07:11:48.744 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-23T07:11:48.745 INFO:tasks.ceph.osd.2:Started 2023-09-23T07:11:48.746 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T07:11:48.822 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:48.821+0000 7fea2576b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.823 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:48.821+0000 7f07eb7c8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.826 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:48.825+0000 7f07eb7c8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.826 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:48.825+0000 7f07eb7c8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.827 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:48.825+0000 7fea2576b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.828 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:48.826+0000 7fea2576b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.842 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:48.841+0000 7fa4c6fe5640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.845 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:48.844+0000 7fa4c6fe5640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.845 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:48.844+0000 7fa4c6fe5640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T07:11:48.972 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:11:48.972 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":5,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:11:48.725951+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:11:48.985 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-23T07:11:48.985 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-23T07:11:49.129 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:49.128+0000 7f55dee71200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-23T07:11:49.287 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T07:11:49.328 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:49.326+0000 7f55dee71200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-23T07:11:49.346 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:49.344+0000 7fea2576b640 -1 Falling back to public interface 2023-09-23T07:11:49.350 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:49.349+0000 7f07eb7c8640 -1 Falling back to public interface 2023-09-23T07:11:49.366 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:49.365+0000 7fa4c6fe5640 -1 Falling back to public interface 2023-09-23T07:11:49.487 INFO:teuthology.misc.health.smithi140.stdout: 2023-09-23T07:11:49.487 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":5,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:11:48.725951+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:11:49.499 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-23T07:11:49.701 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:49.700+0000 7f55dee71200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-23T07:11:50.373 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.372+0000 7f55dee71200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-23T07:11:50.446 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.444+0000 7f55dee71200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-23T07:11:50.581 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.580+0000 7f55dee71200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-23T07:11:50.651 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.650+0000 7f55dee71200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-23T07:11:50.834 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.833+0000 7f55dee71200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-23T07:11:50.914 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.912+0000 7f55dee71200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-23T07:11:50.990 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:50.989+0000 7f55dee71200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-23T07:11:51.089 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:51.088+0000 7f55dee71200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-23T07:11:51.271 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:51.269+0000 7f55dee71200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-23T07:11:51.499 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:51.497+0000 7f55dee71200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-23T07:11:51.654 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:51.653+0000 7f55dee71200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-23T07:11:51.725 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:51.723+0000 7f55dee71200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-23T07:11:52.314 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:52.313+0000 7f55dee71200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-23T07:11:52.646 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:52.644+0000 7f55dee71200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-23T07:11:52.791 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:52.789+0000 7f55dee71200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-23T07:11:52.901 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:52.900+0000 7fea2576b640 -1 osd.0 0 log_to_monitors true 2023-09-23T07:11:52.913 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:52.912+0000 7f07eb7c8640 -1 osd.2 0 log_to_monitors true 2023-09-23T07:11:52.922 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:52.921+0000 7fa4c6fe5640 -1 osd.1 0 log_to_monitors true 2023-09-23T07:11:53.207 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:53.205+0000 7f55dee71200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-23T07:11:53.355 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:53.354+0000 7f55dee71200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-23T07:11:53.508 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:53.507+0000 7f55dee71200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-23T07:11:53.834 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:53.832+0000 7f55dee71200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-23T07:11:55.660 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:55.659+0000 7fea19b0c700 -1 osd.0 0 waiting for initial osdmap 2023-09-23T07:11:55.671 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:11:55.670+0000 7fea116e6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T07:11:55.672 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:55.670+0000 7fa4bb386700 -1 osd.1 0 waiting for initial osdmap 2023-09-23T07:11:55.685 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:11:55.682+0000 7fa4b2f60700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T07:11:55.801 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T07:11:55.854 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:11:55.853+0000 7f55a01e0700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-23T07:11:55.877 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:55.876+0000 7f07dfb69700 -1 osd.2 0 waiting for initial osdmap 2023-09-23T07:11:55.893 INFO:tasks.ceph.osd.2.smithi140.stderr:2023-09-23T07:11:55.892+0000 7f07d7743700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T07:11:56.117 INFO:teuthology.misc.health.smithi140.stdout: 2023-09-23T07:11:56.117 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":7,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:11:54.712462+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:11:56.129 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-23T07:12:02.431 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T07:12:02.758 INFO:teuthology.misc.health.smithi140.stdout: 2023-09-23T07:12:02.758 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":12,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:12:00.867384+0000","last_up_change":"2023-09-23T07:11:56.672362+0000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T07:11:57.892456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6801","nonce":1002728916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6803","nonce":1002728916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6807","nonce":1002728916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6805","nonce":1002728916}]},"public_addr":"172.21.15.140:6801/1002728916","cluster_addr":"172.21.15.140:6803/1002728916","heartbeat_back_addr":"172.21.15.140:6807/1002728916","heartbeat_front_addr":"172.21.15.140:6805/1002728916","state":["exists","up"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6816","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6817","nonce":155684008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6819","nonce":155684008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6823","nonce":155684008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6820","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6821","nonce":155684008}]},"public_addr":"172.21.15.140:6817/155684008","cluster_addr":"172.21.15.140:6819/155684008","heartbeat_back_addr":"172.21.15.140:6823/155684008","heartbeat_front_addr":"172.21.15.140:6821/155684008","state":["exists","up"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6809","nonce":4141700485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6811","nonce":4141700485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6815","nonce":4141700485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6812","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6813","nonce":4141700485}]},"public_addr":"172.21.15.140:6809/4141700485","cluster_addr":"172.21.15.140:6811/4141700485","heartbeat_back_addr":"172.21.15.140:6815/4141700485","heartbeat_front_addr":"172.21.15.140:6813/4141700485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.943415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.880737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.948563+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:12:02.771 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-23T07:12:02.771 INFO:tasks.ceph:Creating RBD pool 2023-09-23T07:12:02.771 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-23T07:12:03.886 INFO:teuthology.orchestra.run.smithi140.stderr:pool 'rbd' created 2023-09-23T07:12:03.901 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-23T07:12:04.886 INFO:teuthology.orchestra.run.smithi140.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-23T07:12:04.903 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-23T07:12:04.904 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-23T07:12:04.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-23T07:12:04.904 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-23T07:12:04.904 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-23T07:12:05.284 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:05.297 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6824","nonce":1839977731},{"type":"v1","addr":"172.21.15.140:6825","nonce":1839977731}]},"active_addr":"172.21.15.140:6825/1839977731","active_change":"2023-09-23T07:11:53.847472+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.140:0","nonce":279050293}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.140:0","nonce":3280948688}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.140:0","nonce":1299976189}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.140:0","nonce":325157858}]}]} 2023-09-23T07:12:05.300 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-23T07:12:05.300 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-23T07:12:05.300 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T07:12:05.618 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:05.618 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":14,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:12:04.882046+0000","last_up_change":"2023-09-23T07:11:56.672362+0000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T07:11:57.892456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T07:12:03.106437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6801","nonce":1002728916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6803","nonce":1002728916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6807","nonce":1002728916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6805","nonce":1002728916}]},"public_addr":"172.21.15.140:6801/1002728916","cluster_addr":"172.21.15.140:6803/1002728916","heartbeat_back_addr":"172.21.15.140:6807/1002728916","heartbeat_front_addr":"172.21.15.140:6805/1002728916","state":["exists","up"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6816","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6817","nonce":155684008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6819","nonce":155684008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6823","nonce":155684008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6820","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6821","nonce":155684008}]},"public_addr":"172.21.15.140:6817/155684008","cluster_addr":"172.21.15.140:6819/155684008","heartbeat_back_addr":"172.21.15.140:6823/155684008","heartbeat_front_addr":"172.21.15.140:6821/155684008","state":["exists","up"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6809","nonce":4141700485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6811","nonce":4141700485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6815","nonce":4141700485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6812","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6813","nonce":4141700485}]},"public_addr":"172.21.15.140:6809/4141700485","cluster_addr":"172.21.15.140:6811/4141700485","heartbeat_back_addr":"172.21.15.140:6815/4141700485","heartbeat_front_addr":"172.21.15.140:6813/4141700485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.943415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.880737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.948563+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:12:05.632 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-23T07:12:05.632 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T07:12:05.952 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:05.953 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":15,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T07:12:05.891638+0000","last_up_change":"2023-09-23T07:11:56.672362+0000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T07:11:57.892456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T07:12:03.106437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6801","nonce":1002728916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6803","nonce":1002728916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6807","nonce":1002728916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6805","nonce":1002728916}]},"public_addr":"172.21.15.140:6801/1002728916","cluster_addr":"172.21.15.140:6803/1002728916","heartbeat_back_addr":"172.21.15.140:6807/1002728916","heartbeat_front_addr":"172.21.15.140:6805/1002728916","state":["exists","up"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6816","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6817","nonce":155684008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6819","nonce":155684008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6823","nonce":155684008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6820","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6821","nonce":155684008}]},"public_addr":"172.21.15.140:6817/155684008","cluster_addr":"172.21.15.140:6819/155684008","heartbeat_back_addr":"172.21.15.140:6823/155684008","heartbeat_front_addr":"172.21.15.140:6821/155684008","state":["exists","up"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6809","nonce":4141700485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6811","nonce":4141700485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6815","nonce":4141700485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6812","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6813","nonce":4141700485}]},"public_addr":"172.21.15.140:6809/4141700485","cluster_addr":"172.21.15.140:6811/4141700485","heartbeat_back_addr":"172.21.15.140:6815/4141700485","heartbeat_front_addr":"172.21.15.140:6813/4141700485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.943415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.880737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.948563+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T07:12:05.967 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-23T07:12:05.967 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-23T07:12:05.968 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-23T07:12:06.138 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-09-23T07:12:06.139 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-23T07:12:06.148 INFO:teuthology.orchestra.run.smithi140.stdout:34359738372 2023-09-23T07:12:06.148 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-23T07:12:06.149 INFO:teuthology.orchestra.run.smithi140.stdout:34359738372 2023-09-23T07:12:06.150 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-23T07:12:06.496 INFO:teuthology.orchestra.run.smithi140.stdout:34359738372 2023-09-23T07:12:06.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-09-23T07:12:06.535 INFO:teuthology.orchestra.run.smithi140.stdout:34359738371 2023-09-23T07:12:06.548 INFO:teuthology.orchestra.run.smithi140.stdout:34359738371 2023-09-23T07:12:06.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-09-23T07:12:06.562 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-09-23T07:12:07.512 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-23T07:12:07.550 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-23T07:12:07.563 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-23T07:12:07.887 INFO:teuthology.orchestra.run.smithi140.stdout:34359738372 2023-09-23T07:12:07.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-09-23T07:12:07.916 INFO:teuthology.orchestra.run.smithi140.stdout:34359738371 2023-09-23T07:12:07.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-09-23T07:12:07.943 INFO:teuthology.orchestra.run.smithi140.stdout:34359738371 2023-09-23T07:12:07.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-09-23T07:12:08.902 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-23T07:12:08.931 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-23T07:12:08.957 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-23T07:12:09.258 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-09-23T07:12:09.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738374 for osd.1 2023-09-23T07:12:09.272 DEBUG:teuthology.parallel:result is None 2023-09-23T07:12:09.319 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-09-23T07:12:09.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.0 2023-09-23T07:12:09.334 DEBUG:teuthology.parallel:result is None 2023-09-23T07:12:09.360 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-09-23T07:12:09.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2023-09-23T07:12:09.374 DEBUG:teuthology.parallel:result is None 2023-09-23T07:12:09.374 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-23T07:12:09.374 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T07:12:09.730 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:09.732 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T07:12:09.743 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-09-23T07:12:07.853791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81544,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033976,"statfs":{"total":289910292480,"available":289826791424,"internally_reserved":0,"allocated":491520,"data_stored":1285374,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000613"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903357+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T07:12:05.903357+0000","last_peered":"2023-09-23T07:12:05.903357+0000","last_clean":"2023-09-23T07:12:05.903357+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T07:12:05.903357+0000","last_undegraded":"2023-09-23T07:12:05.903357+0000","last_fullsized":"2023-09-23T07:12:05.903357+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:22:47.585270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903154+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T07:12:05.903154+0000","last_peered":"2023-09-23T07:12:05.903154+0000","last_clean":"2023-09-23T07:12:05.903154+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T07:12:05.903154+0000","last_undegraded":"2023-09-23T07:12:05.903154+0000","last_fullsized":"2023-09-23T07:12:05.903154+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:46.793786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903274+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T07:12:05.903274+0000","last_peered":"2023-09-23T07:12:05.903274+0000","last_clean":"2023-09-23T07:12:05.903274+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T07:12:05.903274+0000","last_undegraded":"2023-09-23T07:12:05.903274+0000","last_fullsized":"2023-09-23T07:12:05.903274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:45:34.095014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.900910+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T07:12:04.900910+0000","last_peered":"2023-09-23T07:12:04.900910+0000","last_clean":"2023-09-23T07:12:04.900910+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T07:12:04.900910+0000","last_undegraded":"2023-09-23T07:12:04.900910+0000","last_fullsized":"2023-09-23T07:12:04.900910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:41:03.957961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.892636+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T07:12:04.892636+0000","last_peered":"2023-09-23T07:12:04.892636+0000","last_clean":"2023-09-23T07:12:04.892636+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T07:12:04.892636+0000","last_undegraded":"2023-09-23T07:12:04.892636+0000","last_fullsized":"2023-09-23T07:12:04.892636+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:01:08.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903155+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T07:12:05.903155+0000","last_peered":"2023-09-23T07:12:05.903155+0000","last_clean":"2023-09-23T07:12:05.903155+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T07:12:05.903155+0000","last_undegraded":"2023-09-23T07:12:05.903155+0000","last_fullsized":"2023-09-23T07:12:05.903155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:02:47.322360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.900907+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T07:12:04.900907+0000","last_peered":"2023-09-23T07:12:04.900907+0000","last_clean":"2023-09-23T07:12:04.900907+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T07:12:04.900907+0000","last_undegraded":"2023-09-23T07:12:04.900907+0000","last_fullsized":"2023-09-23T07:12:04.900907+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:50:04.647624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.903546+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T07:12:04.903546+0000","last_peered":"2023-09-23T07:12:04.903546+0000","last_clean":"2023-09-23T07:12:04.903546+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T07:12:04.903546+0000","last_undegraded":"2023-09-23T07:12:04.903546+0000","last_fullsized":"2023-09-23T07:12:04.903546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:12:28.643961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903214+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T07:12:05.903214+0000","last_peered":"2023-09-23T07:12:05.903214+0000","last_clean":"2023-09-23T07:12:05.903214+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T07:12:05.903214+0000","last_undegraded":"2023-09-23T07:12:05.903214+0000","last_fullsized":"2023-09-23T07:12:05.903214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:45.472429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":627566,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T07:12:09.744 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T07:12:10.058 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:10.060 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T07:12:10.071 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-09-23T07:12:09.854072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81544,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033976,"statfs":{"total":289910292480,"available":289826791424,"internally_reserved":0,"allocated":491520,"data_stored":1285374,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000894"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903357+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T07:12:05.903357+0000","last_peered":"2023-09-23T07:12:05.903357+0000","last_clean":"2023-09-23T07:12:05.903357+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T07:12:05.903357+0000","last_undegraded":"2023-09-23T07:12:05.903357+0000","last_fullsized":"2023-09-23T07:12:05.903357+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:22:47.585270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903154+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T07:12:05.903154+0000","last_peered":"2023-09-23T07:12:05.903154+0000","last_clean":"2023-09-23T07:12:05.903154+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T07:12:05.903154+0000","last_undegraded":"2023-09-23T07:12:05.903154+0000","last_fullsized":"2023-09-23T07:12:05.903154+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:46.793786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903274+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T07:12:05.903274+0000","last_peered":"2023-09-23T07:12:05.903274+0000","last_clean":"2023-09-23T07:12:05.903274+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T07:12:05.903274+0000","last_undegraded":"2023-09-23T07:12:05.903274+0000","last_fullsized":"2023-09-23T07:12:05.903274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:45:34.095014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.900910+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T07:12:04.900910+0000","last_peered":"2023-09-23T07:12:04.900910+0000","last_clean":"2023-09-23T07:12:04.900910+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T07:12:04.900910+0000","last_undegraded":"2023-09-23T07:12:04.900910+0000","last_fullsized":"2023-09-23T07:12:04.900910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:41:03.957961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.892636+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T07:12:04.892636+0000","last_peered":"2023-09-23T07:12:04.892636+0000","last_clean":"2023-09-23T07:12:04.892636+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T07:12:04.892636+0000","last_undegraded":"2023-09-23T07:12:04.892636+0000","last_fullsized":"2023-09-23T07:12:04.892636+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:01:08.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903155+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T07:12:05.903155+0000","last_peered":"2023-09-23T07:12:05.903155+0000","last_clean":"2023-09-23T07:12:05.903155+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T07:12:05.903155+0000","last_undegraded":"2023-09-23T07:12:05.903155+0000","last_fullsized":"2023-09-23T07:12:05.903155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:02:47.322360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.900907+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T07:12:04.900907+0000","last_peered":"2023-09-23T07:12:04.900907+0000","last_clean":"2023-09-23T07:12:04.900907+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T07:12:04.900907+0000","last_undegraded":"2023-09-23T07:12:04.900907+0000","last_fullsized":"2023-09-23T07:12:04.900907+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:50:04.647624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T07:12:04.903546+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T07:12:04.903546+0000","last_peered":"2023-09-23T07:12:04.903546+0000","last_clean":"2023-09-23T07:12:04.903546+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T07:12:04.903546+0000","last_undegraded":"2023-09-23T07:12:04.903546+0000","last_fullsized":"2023-09-23T07:12:04.903546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:12:28.643961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T07:12:05.903214+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T07:12:05.903214+0000","last_peered":"2023-09-23T07:12:05.903214+0000","last_clean":"2023-09-23T07:12:05.903214+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T07:12:05.903214+0000","last_undegraded":"2023-09-23T07:12:05.903214+0000","last_fullsized":"2023-09-23T07:12:05.903214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:45.472429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":627566,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T07:12:10.073 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-23T07:12:10.073 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-23T07:12:10.073 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-23T07:12:10.073 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-23T07:12:10.433 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:10.433 INFO:teuthology.orchestra.run.smithi140.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-23T07:12:10.447 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-09-23T07:12:10.447 INFO:teuthology.run_tasks:Running task exec... 2023-09-23T07:12:10.456 INFO:teuthology.task.exec:Executing custom commands... 2023-09-23T07:12:10.456 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T07:12:10.456 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2' 2023-09-23T07:12:11.740 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile' 2023-09-23T07:12:13.734 INFO:teuthology.orchestra.run.smithi140.stderr:pool 'datapool' created 2023-09-23T07:12:13.749 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set datapool allow_ec_overwrites true' 2023-09-23T07:12:14.737 INFO:teuthology.orchestra.run.smithi140.stderr:set pool 3 allow_ec_overwrites to true 2023-09-23T07:12:14.754 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-09-23T07:12:17.773 INFO:teuthology.run_tasks:Running task install... 2023-09-23T07:12:17.781 DEBUG:teuthology.task.install:project ceph 2023-09-23T07:12:17.781 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}} 2023-09-23T07:12:17.781 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'} 2023-09-23T07:12:17.781 INFO:teuthology.task.install:Using flavor: default 2023-09-23T07:12:17.786 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-23T07:12:17.786 INFO:teuthology.task.install:extra packages: [] 2023-09-23T07:12:17.787 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['fio'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'tag': None, 'wait_for_package': False} 2023-09-23T07:12:17.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:12:18.135 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/ 2023-09-23T07:12:18.135 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6334.gee696c23 2023-09-23T07:12:18.376 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-23T07:12:18.376 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:12:18.376 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-23T07:12:18.410 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, fio, fio on remote rpm x86_64 2023-09-23T07:12:18.410 DEBUG:teuthology.orchestra.run.smithi140:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-23T07:12:18.482 DEBUG:teuthology.orchestra.run.smithi140:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-23T07:12:18.552 DEBUG:teuthology.orchestra.run.smithi140:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-23T07:12:18.610 INFO:teuthology.orchestra.run.smithi140.stdout:check_obsoletes = 1 2023-09-23T07:12:18.623 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum clean all 2023-09-23T07:12:18.895 INFO:teuthology.orchestra.run.smithi140.stdout:71 files removed 2023-09-23T07:12:18.923 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd fio fio 2023-09-23T07:12:19.704 INFO:teuthology.orchestra.run.smithi140.stdout:ceph packages for x86_64 223 kB/s | 81 kB 00:00 2023-09-23T07:12:20.088 INFO:teuthology.orchestra.run.smithi140.stdout:ceph noarch packages 27 kB/s | 9.4 kB 00:00 2023-09-23T07:12:20.480 INFO:teuthology.orchestra.run.smithi140.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-09-23T07:12:22.398 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - AppStream 18 MB/s | 33 MB 00:01 2023-09-23T07:12:31.407 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 47 MB 00:02 2023-09-23T07:12:38.565 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2023-09-23T07:12:39.157 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-23T07:12:43.342 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS Stream 8 - PowerTools 1.6 MB/s | 6.5 MB 00:04 2023-09-23T07:12:44.810 INFO:teuthology.orchestra.run.smithi140.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-23T07:12:46.394 INFO:teuthology.orchestra.run.smithi140.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-09-23T07:12:50.194 INFO:teuthology.orchestra.run.smithi140.stdout:lab-extras 372 kB/s | 24 kB 00:00 2023-09-23T07:12:54.409 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-radosgw-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.410 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-test-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.411 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.412 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-base-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.413 INFO:teuthology.orchestra.run.smithi140.stdout:Package cephadm-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.415 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-immutable-object-cache-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.415 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.416 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-dashboard-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.417 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-diskprediction-local-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.419 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-rook-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.419 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-mgr-cephadm-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.421 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.422 INFO:teuthology.orchestra.run.smithi140.stdout:Package ceph-volume-2:18.0.0-6334.gee696c23.el8.noarch is already installed. 2023-09-23T07:12:54.423 INFO:teuthology.orchestra.run.smithi140.stdout:Package librados-devel-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.424 INFO:teuthology.orchestra.run.smithi140.stdout:Package libcephfs2-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.425 INFO:teuthology.orchestra.run.smithi140.stdout:Package libcephfs-devel-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.426 INFO:teuthology.orchestra.run.smithi140.stdout:Package librados2-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.427 INFO:teuthology.orchestra.run.smithi140.stdout:Package librbd1-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.428 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rados-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.429 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rgw-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.430 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-cephfs-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.431 INFO:teuthology.orchestra.run.smithi140.stdout:Package python3-rbd-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.432 INFO:teuthology.orchestra.run.smithi140.stdout:Package rbd-fuse-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.433 INFO:teuthology.orchestra.run.smithi140.stdout:Package rbd-mirror-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.434 INFO:teuthology.orchestra.run.smithi140.stdout:Package rbd-nbd-2:18.0.0-6334.gee696c23.el8.x86_64 is already installed. 2023-09-23T07:12:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout: Package Architecture Version Repository Size 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout: fio x86_64 3.19-4.el8 appstream 538 k 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-09-23T07:12:54.617 INFO:teuthology.orchestra.run.smithi140.stdout: libpmemblk x86_64 1.6.1-1.el8 appstream 92 k 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout:Install 4 Packages 2023-09-23T07:12:54.618 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:54.619 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 752 k 2023-09-23T07:12:54.619 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 2.5 M 2023-09-23T07:12:54.619 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2023-09-23T07:12:54.929 INFO:teuthology.orchestra.run.smithi140.stdout:(1/4): daxctl-libs-71.1-7.el8.x86_64.rpm 220 kB/s | 42 kB 00:00 2023-09-23T07:12:54.963 INFO:teuthology.orchestra.run.smithi140.stdout:(2/4): libpmemblk-1.6.1-1.el8.x86_64.rpm 407 kB/s | 92 kB 00:00 2023-09-23T07:12:55.030 INFO:teuthology.orchestra.run.smithi140.stdout:(3/4): ndctl-libs-71.1-7.el8.x86_64.rpm 799 kB/s | 80 kB 00:00 2023-09-23T07:12:55.080 INFO:teuthology.orchestra.run.smithi140.stdout:(4/4): fio-3.19-4.el8.x86_64.rpm 1.5 MB/s | 538 kB 00:00 2023-09-23T07:12:55.080 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2023-09-23T07:12:55.080 INFO:teuthology.orchestra.run.smithi140.stdout:Total 1.6 MB/s | 752 kB 00:00 2023-09-23T07:12:55.119 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2023-09-23T07:12:55.131 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2023-09-23T07:12:55.132 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2023-09-23T07:12:55.200 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2023-09-23T07:12:55.203 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2023-09-23T07:12:55.405 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2023-09-23T07:12:55.642 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 1/4 2023-09-23T07:12:55.723 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 1/4 2023-09-23T07:12:55.959 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 2/4 2023-09-23T07:12:56.010 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 2/4 2023-09-23T07:12:56.226 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-09-23T07:12:56.364 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-09-23T07:12:56.625 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : fio-3.19-4.el8.x86_64 4/4 2023-09-23T07:12:56.923 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: fio-3.19-4.el8.x86_64 4/4 2023-09-23T07:12:56.923 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : fio-3.19-4.el8.x86_64 1/4 2023-09-23T07:12:56.923 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : libpmemblk-1.6.1-1.el8.x86_64 2/4 2023-09-23T07:12:56.923 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 3/4 2023-09-23T07:12:57.606 INFO:teuthology.orchestra.run.smithi140.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 4/4 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout:Installed: 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout: daxctl-libs-71.1-7.el8.x86_64 fio-3.19-4.el8.x86_64 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout: libpmemblk-1.6.1-1.el8.x86_64 ndctl-libs-71.1-7.el8.x86_64 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T07:12:57.607 INFO:teuthology.orchestra.run.smithi140.stdout:Complete! 2023-09-23T07:12:57.942 DEBUG:teuthology.parallel:result is None 2023-09-23T07:12:57.942 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:12:58.253 DEBUG:teuthology.orchestra.run.smithi140:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-23T07:12:58.275 INFO:teuthology.orchestra.run.smithi140.stdout:18.0.0-6334.gee696c23.el8 2023-09-23T07:12:58.276 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6334.gee696c23.el8 2023-09-23T07:12:58.276 INFO:teuthology.task.install:The correct ceph version 18.0.0-6334.gee696c23 is installed. 2023-09-23T07:12:58.276 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-09-23T07:12:58.276 INFO:teuthology.run_tasks:Running task workunit... 2023-09-23T07:12:58.288 INFO:tasks.workunit:Pulling workunits from ref ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:12:58.288 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-23T07:12:58.288 DEBUG:teuthology.orchestra.run.smithi140:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-23T07:12:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T07:12:58.334 INFO:teuthology.orchestra.run.smithi140.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-23T07:12:58.334 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-23T07:12:58.392 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-23T07:12:58.392 DEBUG:teuthology.orchestra.run.smithi140:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-23T07:12:58.450 INFO:tasks.workunit:timeout=3h 2023-09-23T07:12:58.450 INFO:tasks.workunit:cleanup=True 2023-09-23T07:12:58.450 DEBUG:teuthology.orchestra.run.smithi140:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T07:12:58.509 INFO:tasks.workunit.client.0.smithi140.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-23T07:13:58.824 INFO:tasks.workunit.client.0.smithi140.stderr:Updating files: 96% (11569/11967) Updating files: 97% (11608/11967) Updating files: 98% (11728/11967) Updating files: 99% (11848/11967) Updating files: 100% (11967/11967) Updating files: 100% (11967/11967), done. 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:Note: switching to 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'. 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:state without impacting any branches by switching back to a branch. 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-23T07:13:59.474 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: git switch -c 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr:Or undo this operation with: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: git switch - 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr: 2023-09-23T07:13:59.475 INFO:tasks.workunit.client.0.smithi140.stderr:HEAD is now at ee696c23a60 qa/suites/rbd: add test to check rbd_support module recovery 2023-09-23T07:13:59.478 DEBUG:teuthology.orchestra.run.smithi140:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-23T07:13:59.540 INFO:tasks.workunit.client.0.smithi140.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-23T07:13:59.543 INFO:tasks.workunit.client.0.smithi140.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-23T07:13:59.543 INFO:tasks.workunit.client.0.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-23T07:14:00.091 INFO:tasks.workunit.client.0.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-23T07:14:00.155 INFO:tasks.workunit.client.0.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-23T07:14:00.193 INFO:tasks.workunit.client.0.smithi140.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-23T07:14:00.194 INFO:tasks.workunit.client.0.smithi140.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-23T07:14:00.194 INFO:tasks.workunit.client.0.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-23T07:14:00.231 INFO:tasks.workunit.client.0.smithi140.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-23T07:14:00.234 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-09-23T07:14:00.234 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-23T07:14:00.292 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-09-23T07:14:00.293 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-09-23T07:14:00.294 DEBUG:teuthology.orchestra.run.smithi140:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ POOL=rbd 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ IMAGE_PREFIX=image 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ MNT_DIR_PREFIX=mnt 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ NUM_IMAGES=20 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ RUN_TIME=3600 2023-09-23T07:14:00.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror pool enable rbd image 2023-09-23T07:14:00.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror pool peer add rbd dummy 2023-09-23T07:14:00.426 INFO:tasks.workunit.client.0.smithi140.stdout:6e5d5e76-e90c-4f2a-a6a0-9a5984e5deba 2023-09-23T07:14:00.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i = 1 )) 2023-09-23T07:14:00.429 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:00.429 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image1 2023-09-23T07:14:00.472 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-09-23T07:14:00.992 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:00.998 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-09-23T07:14:01.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:01.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:01.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image2 2023-09-23T07:14:01.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-09-23T07:14:01.997 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:02.007 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-09-23T07:14:02.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:02.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:02.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image3 2023-09-23T07:14:02.109 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-09-23T07:14:03.005 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:03.013 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-09-23T07:14:03.064 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:03.064 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:03.064 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image4 2023-09-23T07:14:03.108 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-09-23T07:14:04.020 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:04.030 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-09-23T07:14:04.085 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:04.085 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:04.085 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image5 2023-09-23T07:14:04.130 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-09-23T07:14:05.017 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:05.024 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-09-23T07:14:05.075 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:05.075 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:05.075 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image6 2023-09-23T07:14:05.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-09-23T07:14:06.022 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:06.030 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-09-23T07:14:06.083 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:06.083 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:06.083 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image7 2023-09-23T07:14:06.127 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-09-23T07:14:07.029 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:07.040 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-09-23T07:14:07.091 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:07.091 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:07.091 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image8 2023-09-23T07:14:07.137 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-09-23T07:14:08.031 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:08.040 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-09-23T07:14:08.092 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:08.092 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:08.092 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image9 2023-09-23T07:14:08.138 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-09-23T07:14:09.032 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:09.040 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-09-23T07:14:09.092 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:09.093 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:09.093 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image10 2023-09-23T07:14:09.137 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-09-23T07:14:10.035 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:10.042 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-09-23T07:14:10.095 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:10.095 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:10.095 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image11 2023-09-23T07:14:10.140 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-09-23T07:14:11.041 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:11.051 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-09-23T07:14:11.103 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:11.104 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:11.104 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image12 2023-09-23T07:14:11.148 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-09-23T07:14:12.042 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:12.053 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-09-23T07:14:12.106 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:12.106 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:12.106 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image13 2023-09-23T07:14:12.153 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-09-23T07:14:13.045 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:13.052 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-09-23T07:14:13.103 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:13.103 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:13.103 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image14 2023-09-23T07:14:13.146 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-09-23T07:14:14.050 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:14.057 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-09-23T07:14:14.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:14.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:14.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image15 2023-09-23T07:14:14.154 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-09-23T07:14:15.052 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:15.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-09-23T07:14:15.112 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:15.112 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:15.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image16 2023-09-23T07:14:15.158 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-09-23T07:14:16.056 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:16.063 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-09-23T07:14:16.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:16.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:16.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image17 2023-09-23T07:14:16.162 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-09-23T07:14:17.059 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:17.067 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-09-23T07:14:17.120 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:17.120 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:17.120 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image18 2023-09-23T07:14:17.165 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-09-23T07:14:18.064 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:18.071 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-09-23T07:14:18.124 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:18.124 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:18.124 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image19 2023-09-23T07:14:18.168 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-09-23T07:14:19.065 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:19.073 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-09-23T07:14:19.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:19.126 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:19.126 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd create --size 1G rbd/image20 2023-09-23T07:14:19.169 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-09-23T07:14:20.073 INFO:tasks.workunit.client.0.smithi140.stdout:Mirroring enabled 2023-09-23T07:14:20.080 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-09-23T07:14:20.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i = 1 )) 2023-09-23T07:14:20.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.133 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image1 2023-09-23T07:14:20.287 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd0 2023-09-23T07:14:20.287 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.287 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.287 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.287 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image2 2023-09-23T07:14:20.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd1 2023-09-23T07:14:20.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.354 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.355 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image3 2023-09-23T07:14:20.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd2 2023-09-23T07:14:20.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.427 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image4 2023-09-23T07:14:20.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd3 2023-09-23T07:14:20.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.504 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.504 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.504 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image5 2023-09-23T07:14:20.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd4 2023-09-23T07:14:20.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.621 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image6 2023-09-23T07:14:20.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd5 2023-09-23T07:14:20.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.765 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image7 2023-09-23T07:14:20.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.940 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd6 2023-09-23T07:14:20.940 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:20.941 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:20.941 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:20.941 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image8 2023-09-23T07:14:21.136 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd7 2023-09-23T07:14:21.136 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:21.136 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:21.136 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:21.144 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image9 2023-09-23T07:14:21.333 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd8 2023-09-23T07:14:21.333 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:21.333 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:21.339 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image10 2023-09-23T07:14:21.353 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:21.676 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd9 2023-09-23T07:14:21.676 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:21.676 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:21.700 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image11 2023-09-23T07:14:22.084 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd10 2023-09-23T07:14:22.084 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:22.084 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:22.088 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:22.095 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image12 2023-09-23T07:14:22.585 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd11 2023-09-23T07:14:22.585 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:22.586 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:22.586 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:22.586 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image13 2023-09-23T07:14:22.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd12 2023-09-23T07:14:22.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:22.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:22.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:22.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image14 2023-09-23T07:14:23.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd13 2023-09-23T07:14:23.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:23.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:23.320 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:23.326 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image15 2023-09-23T07:14:23.615 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd14 2023-09-23T07:14:23.616 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:23.616 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:23.616 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:23.616 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image16 2023-09-23T07:14:24.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd15 2023-09-23T07:14:24.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:24.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:24.030 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:24.035 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image17 2023-09-23T07:14:24.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd16 2023-09-23T07:14:24.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:24.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:24.485 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image18 2023-09-23T07:14:24.502 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:24.870 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd17 2023-09-23T07:14:24.870 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:24.871 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:24.871 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image19 2023-09-23T07:14:24.878 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:25.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd18 2023-09-23T07:14:25.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:25.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:25.444 INFO:tasks.workunit.client.0.smithi140.stderr:++ sudo rbd device map rbd/image20 2023-09-23T07:14:25.453 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:25.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ DEVS[$i]=/dev/rbd19 2023-09-23T07:14:25.796 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T07:14:25.796 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 20 )) 2023-09-23T07:14:25.796 INFO:tasks.workunit.client.0.smithi140.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T07:14:25.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453265 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ END_TIME=1695456865 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT= 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR= 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453265 -le 1695456865 ]] 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z '' ]] 2023-09-23T07:14:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:14:35.822 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:14:35.822 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:14:35.822 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:14:35.854 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:14:36.908 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1299976189 2023-09-23T07:14:36.908 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:14:36.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453276 2023-09-23T07:14:36.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453276 -le 1695456865 ]] 2023-09-23T07:14:36.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1299976189 ]] 2023-09-23T07:14:36.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1299976189 != '' ]] 2023-09-23T07:14:36.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/1299976189 2023-09-23T07:14:39.026 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/1299976189 until 2023-09-23T08:14:38.267311+0000 (3600 sec) 2023-09-23T07:14:39.051 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:14:39.051 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/1299976189 2023-09-23T07:14:39.922 INFO:tasks.workunit.client.0.smithi140.stderr:listed 1 entries 2023-09-23T07:14:39.939 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/1299976189 2023-09-23T07:14:39.939 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:14:49.942 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:14:49.942 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:14:49.942 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:14:49.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:14:51.043 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1299976189 2023-09-23T07:14:51.044 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:14:51.044 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453291 2023-09-23T07:14:51.044 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453291 -le 1695456865 ]] 2023-09-23T07:14:51.045 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1299976189 ]] 2023-09-23T07:14:51.045 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1299976189 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\2\9\9\9\7\6\1\8\9 ]] 2023-09-23T07:14:51.045 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:01.049 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:01.049 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:01.050 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:15:01.055 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:01.650 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:15:01.649+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad8137ee0 2023-09-23T07:15:01.650 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:15:01.649+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badfaf7a00 2023-09-23T07:15:01.650 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:15:01.649+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bae2fe4000 2023-09-23T07:15:01.650 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:15:01.649+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bae412f930 2023-09-23T07:15:01.651 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:15:01.650+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bae4ced790 2023-09-23T07:15:02.057 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:15:02.058 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:15:02.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453302 2023-09-23T07:15:02.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453302 -le 1695456865 ]] 2023-09-23T07:15:02.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:15:02.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\2\9\9\9\7\6\1\8\9 ]] 2023-09-23T07:15:02.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/2641832162 2023-09-23T07:15:03.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:03.047+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3bad8c600 image watch failed: 94161698330624, (107) Transport endpoint is not connected 2023-09-23T07:15:03.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:03.047+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bad8c600 handle_error: handle=94161698330624: (107) Transport endpoint is not connected 2023-09-23T07:15:03.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:03.048+0000 7f5568565700 -1 librbd::watcher::RewatchRequest: 0x55a3bad53900 handle_unwatch client blocklisted 2023-09-23T07:15:03.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:03.048+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bad8c600 handle_rewatch: client blocklisted 2023-09-23T07:15:04.049 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.048+0000 7f5568565700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.048+0000 7f5568565700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3bacd64e0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.048+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb06dce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.050+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.050+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb18b340 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.050+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3baf30540 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.050+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:15:04.051+0000 7f5568565700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3bac57380 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:15:04.058 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/2641832162 until 2023-09-23T08:15:03.267479+0000 (3600 sec) 2023-09-23T07:15:04.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:15:04.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/2641832162 2023-09-23T07:15:04.839 INFO:tasks.workunit.client.0.smithi140.stderr:listed 2 entries 2023-09-23T07:15:04.854 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/2641832162 2023-09-23T07:15:04.854 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:14.856 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:14.856 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:15:14.857 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:14.857 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:15.815 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:15:15.815 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:15:15.816 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453315 2023-09-23T07:15:15.816 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453315 -le 1695456865 ]] 2023-09-23T07:15:15.816 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:15:15.817 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:15:15.817 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:25.818 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:15:25.820 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:25.823 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:25.823 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453326 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453326 -le 1695456865 ]] 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:15:26.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:36.816 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:36.816 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:36.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:15:36.832 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:37.829 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:15:37.829 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:15:37.830 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453337 2023-09-23T07:15:37.830 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453337 -le 1695456865 ]] 2023-09-23T07:15:37.830 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:15:37.830 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:15:37.831 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:47.840 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:47.840 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:15:47.843 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:47.844 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:49.122 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:15:49.123 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:15:49.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453349 2023-09-23T07:15:49.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453349 -le 1695456865 ]] 2023-09-23T07:15:49.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:15:49.132 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:15:49.132 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:15:59.159 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:15:59.160 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:15:59.160 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:15:59.160 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:16:02.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2641832162 2023-09-23T07:16:02.119 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:16:02.155 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453362 2023-09-23T07:16:02.155 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453362 -le 1695456865 ]] 2023-09-23T07:16:02.155 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2641832162 ]] 2023-09-23T07:16:02.155 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2641832162 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:16:02.155 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:16:12.163 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:16:12.164 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:16:12.164 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:16:12.164 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:16:13.418 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:16:13.420 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:16:13.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453373 2023-09-23T07:16:13.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453373 -le 1695456865 ]] 2023-09-23T07:16:13.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:16:13.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\1\8\3\2\1\6\2 ]] 2023-09-23T07:16:13.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/896161905 2023-09-23T07:16:15.852 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/896161905 until 2023-09-23T08:16:15.075541+0000 (3600 sec) 2023-09-23T07:16:15.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/896161905 2023-09-23T07:16:15.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:16:16.782 INFO:tasks.workunit.client.0.smithi140.stderr:listed 3 entries 2023-09-23T07:16:16.794 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/896161905 2023-09-23T07:16:16.794 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:16:26.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:16:26.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:16:26.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:16:26.806 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:16:28.231 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:16:28.231 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:16:28.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453388 2023-09-23T07:16:28.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453388 -le 1695456865 ]] 2023-09-23T07:16:28.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:16:28.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:16:28.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:16:38.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:16:38.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:16:38.248 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:16:38.248 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:16:39.312 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:16:39.314 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:16:39.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453399 2023-09-23T07:16:39.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453399 -le 1695456865 ]] 2023-09-23T07:16:39.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:16:39.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:16:39.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:16:49.329 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:16:49.329 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:16:49.329 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:16:49.329 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:16:50.378 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:16:50.386 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:16:50.387 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453410 2023-09-23T07:16:50.387 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453410 -le 1695456865 ]] 2023-09-23T07:16:50.388 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:16:50.388 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:16:50.388 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.014+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.014+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb8a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.015+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.015+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.015+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba56fd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.015+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac44000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.037 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.026+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.026+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.026+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac45280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.026+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba031d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb9f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb9900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.042+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb8880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.044+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.044+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.044+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb09c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.044+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac45800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba531000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb8a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba56e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.056+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac45280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.061+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.061+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac44480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.062+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.065 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.062+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba56e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.079 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.077+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.079 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.077+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.079 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.077+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac45300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.079 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.077+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba530100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.099 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.098+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.100 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.098+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.100 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.098+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb9e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.100 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:17:00.098+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb8b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:17:00.393 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:00.393 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:00.398 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:17:00.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:01.413 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:17:01.413 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:17:01.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453421 2023-09-23T07:17:01.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453421 -le 1695456865 ]] 2023-09-23T07:17:01.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:17:01.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:17:01.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:11.418 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:11.418 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:11.418 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:17:11.424 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:12.467 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/896161905 2023-09-23T07:17:12.468 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:17:12.468 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453432 2023-09-23T07:17:12.468 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453432 -le 1695456865 ]] 2023-09-23T07:17:12.468 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/896161905 ]] 2023-09-23T07:17:12.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/896161905 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:17:12.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:22.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:22.482 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:17:22.488 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:22.489 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:23.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:17:23.504 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:17:23.504 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453443 2023-09-23T07:17:23.504 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453443 -le 1695456865 ]] 2023-09-23T07:17:23.505 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:17:23.505 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\8\9\6\1\6\1\9\0\5 ]] 2023-09-23T07:17:23.505 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/9270898 2023-09-23T07:17:23.885 INFO:tasks.workunit.client.0.smithi140.stderr:2023-09-23T07:17:23.884+0000 7ff9bf7fe700 0 --2- 172.21.15.140:0/4275890615 >> v2:172.21.15.140:3300/0 conn(0x7ff9c012adf0 0x7ff9c006c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-23T07:17:26.172 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/9270898 until 2023-09-23T08:17:25.383732+0000 (3600 sec) 2023-09-23T07:17:26.190 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:17:26.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/9270898 2023-09-23T07:17:27.094 INFO:tasks.workunit.client.0.smithi140.stderr:listed 4 entries 2023-09-23T07:17:27.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/9270898 2023-09-23T07:17:27.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:37.117 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:37.121 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:37.121 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:17:37.143 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:38.157 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:17:38.157 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:17:38.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453458 2023-09-23T07:17:38.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453458 -le 1695456865 ]] 2023-09-23T07:17:38.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:17:38.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:17:38.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:48.164 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:48.165 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:48.165 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:17:48.169 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:49.248 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:17:49.248 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:17:49.248 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453469 2023-09-23T07:17:49.249 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453469 -le 1695456865 ]] 2023-09-23T07:17:49.249 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:17:49.249 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:17:49.249 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:17:59.252 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:17:59.252 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:17:59.259 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:17:59.265 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.032+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba530100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bac44200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba56e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.033+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac44b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bac45880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba530100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.061+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba56e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.068+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.068+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.068+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.069+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.069+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.069+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.085 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.080+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.080+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.080+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.080+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.081+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.081+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.081+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.086 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:18:00.081+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac44b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:18:00.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:18:00.312 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:00.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453480 2023-09-23T07:18:00.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453480 -le 1695456865 ]] 2023-09-23T07:18:00.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:18:00.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:18:00.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:18:10.321 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:18:10.321 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:18:10.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:10.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:18:11.390 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:18:11.391 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:11.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453491 2023-09-23T07:18:11.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453491 -le 1695456865 ]] 2023-09-23T07:18:11.392 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:18:11.392 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:18:11.392 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:18:21.393 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:18:21.393 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:18:21.394 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:21.415 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:18:22.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/9270898 2023-09-23T07:18:22.463 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:22.464 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453502 2023-09-23T07:18:22.464 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453502 -le 1695456865 ]] 2023-09-23T07:18:22.464 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/9270898 ]] 2023-09-23T07:18:22.464 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/9270898 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:18:22.464 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:18:32.470 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:18:32.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:18:32.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:18:32.476 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:33.608 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:18:33.608 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:33.609 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453513 2023-09-23T07:18:33.609 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453513 -le 1695456865 ]] 2023-09-23T07:18:33.609 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:18:33.609 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\2\7\0\8\9\8 ]] 2023-09-23T07:18:33.609 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/1813963006 2023-09-23T07:18:35.227 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/1813963006 until 2023-09-23T08:18:34.504483+0000 (3600 sec) 2023-09-23T07:18:35.243 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/1813963006 2023-09-23T07:18:35.243 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:18:36.160 INFO:tasks.workunit.client.0.smithi140.stderr:listed 5 entries 2023-09-23T07:18:36.181 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/1813963006 2023-09-23T07:18:36.181 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:18:46.188 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:18:46.188 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:18:46.188 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:18:46.199 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:47.340 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:18:47.340 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:47.341 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453527 2023-09-23T07:18:47.341 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453527 -le 1695456865 ]] 2023-09-23T07:18:47.341 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:18:47.341 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:18:47.341 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:18:57.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:18:57.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:18:57.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:18:57.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:18:59.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:18:59.013 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:18:59.017 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453539 2023-09-23T07:18:59.017 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453539 -le 1695456865 ]] 2023-09-23T07:18:59.017 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:18:59.017 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:18:59.017 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:19:00.005 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.003+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.005 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.003+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba93fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.006 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.004+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.006 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.004+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb09db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.011 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.010+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.011 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.010+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac45b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.011 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.010+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.010+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb09ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.012+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.012+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac44180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.020+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.020+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.020+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.022 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.020+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.024 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.023+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.024 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.023+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.025+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.025+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3e180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.036 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.034+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac45c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.035+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.035+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.055+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.065+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:19:00.065+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:19:09.019 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:19:09.019 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:19:09.019 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:19:09.020 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:19:10.148 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:19:10.148 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:19:10.149 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453550 2023-09-23T07:19:10.149 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453550 -le 1695456865 ]] 2023-09-23T07:19:10.149 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:19:10.149 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:19:10.149 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:19:20.167 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:19:20.168 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:19:20.168 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:19:20.174 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:19:21.257 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:19:21.257 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:19:21.258 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453561 2023-09-23T07:19:21.258 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453561 -le 1695456865 ]] 2023-09-23T07:19:21.258 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:19:21.258 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:19:21.259 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:19:31.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:19:31.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:19:31.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:19:31.275 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:19:32.333 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1813963006 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453572 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453572 -le 1695456865 ]] 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1813963006 ]] 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1813963006 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:19:32.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:19:42.340 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:19:42.347 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:19:42.347 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:19:42.348 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:19:43.333 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:19:43.336 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:19:43.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453583 2023-09-23T07:19:43.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453583 -le 1695456865 ]] 2023-09-23T07:19:43.338 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:19:43.338 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\8\1\3\9\6\3\0\0\6 ]] 2023-09-23T07:19:43.338 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/947471084 2023-09-23T07:19:46.054 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/947471084 until 2023-09-23T08:19:45.271692+0000 (3600 sec) 2023-09-23T07:19:46.064 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/947471084 2023-09-23T07:19:46.086 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:19:46.976 INFO:tasks.workunit.client.0.smithi140.stderr:listed 6 entries 2023-09-23T07:19:46.992 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/947471084 2023-09-23T07:19:46.992 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:19:57.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:19:57.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:19:57.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:19:57.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:19:58.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:19:58.229 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:19:58.230 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453598 2023-09-23T07:19:58.230 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453598 -le 1695456865 ]] 2023-09-23T07:19:58.230 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:19:58.230 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:19:58.230 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:20:00.028 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.028 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c9100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c9f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.030 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.028+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.028+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.028+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d6b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.028+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.044+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.044+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.044+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.044+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.050+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.051+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.051+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.051+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.065+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.065+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.065+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.065+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.077 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.075+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.077 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.075+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.077 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.075+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.077 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.075+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:20:00.076+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dc100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:20:08.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:20:08.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:20:08.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:20:08.254 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:20:09.290 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:20:09.293 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:20:09.294 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453609 2023-09-23T07:20:09.294 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453609 -le 1695456865 ]] 2023-09-23T07:20:09.295 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:20:09.295 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:20:09.295 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:20:19.302 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:20:19.303 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:20:19.310 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:20:19.314 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:20:20.307 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:20:20.309 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:20:20.309 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453620 2023-09-23T07:20:20.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453620 -le 1695456865 ]] 2023-09-23T07:20:20.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:20:20.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:20:20.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:20:30.315 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:20:30.315 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:20:30.315 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:20:30.319 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:20:31.425 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:20:31.425 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:20:31.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453631 2023-09-23T07:20:31.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453631 -le 1695456865 ]] 2023-09-23T07:20:31.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:20:31.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:20:31.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:20:41.432 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:20:41.432 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:20:41.433 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:20:41.434 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:20:42.526 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/947471084 2023-09-23T07:20:42.527 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:20:42.528 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453642 2023-09-23T07:20:42.528 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453642 -le 1695456865 ]] 2023-09-23T07:20:42.528 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/947471084 ]] 2023-09-23T07:20:42.528 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/947471084 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:20:42.528 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:20:52.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:20:52.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:20:52.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:20:52.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:20:53.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:20:53.745 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:20:53.746 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453653 2023-09-23T07:20:53.746 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453653 -le 1695456865 ]] 2023-09-23T07:20:53.746 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:20:53.746 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\9\4\7\4\7\1\0\8\4 ]] 2023-09-23T07:20:53.747 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/4072506271 2023-09-23T07:20:55.412 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/4072506271 until 2023-09-23T08:20:54.610137+0000 (3600 sec) 2023-09-23T07:20:55.429 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:20:55.429 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/4072506271 2023-09-23T07:20:56.373 INFO:tasks.workunit.client.0.smithi140.stderr:listed 7 entries 2023-09-23T07:20:56.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/4072506271 2023-09-23T07:20:56.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba93e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3baf3e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac44500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3baf3e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.031+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb2c8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dcf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb2c8c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.039+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba93e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.043+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3baf3e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.053 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.053+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.053+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.053+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bac44500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.053+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3baf3e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb2c8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dcf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.063+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.063+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.063+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.063+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb2c8c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.065 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.064+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:00.065 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:21:00.064+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:21:06.401 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:21:06.401 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:21:06.402 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:21:06.405 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:21:07.425 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:21:07.425 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:21:07.426 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453667 2023-09-23T07:21:07.426 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453667 -le 1695456865 ]] 2023-09-23T07:21:07.426 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:21:07.426 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:21:07.426 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:21:08.145 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:21:08.146 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:21:08.144+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:21:17.433 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:21:17.433 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:21:17.445 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:21:17.446 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:21:18.572 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:21:18.573 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:21:18.573 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453678 2023-09-23T07:21:18.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453678 -le 1695456865 ]] 2023-09-23T07:21:18.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:21:18.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:21:18.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:21:28.581 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:21:28.581 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:21:28.584 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:21:28.589 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:21:29.728 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453689 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453689 -le 1695456865 ]] 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:21:29.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:21:39.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:21:39.735 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:21:39.735 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:21:39.735 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:21:40.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:21:40.804 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:21:40.805 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453700 2023-09-23T07:21:40.805 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453700 -le 1695456865 ]] 2023-09-23T07:21:40.805 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:21:40.805 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:21:40.805 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:21:44.289 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:21:44.287+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:21:44.289 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:21:44.288+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:21:50.812 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:21:50.812 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:21:50.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:21:50.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:21:51.949 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4072506271 2023-09-23T07:21:51.949 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:21:51.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453711 2023-09-23T07:21:51.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453711 -le 1695456865 ]] 2023-09-23T07:21:51.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4072506271 ]] 2023-09-23T07:21:51.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4072506271 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:21:51.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:22:01.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:22:01.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:22:01.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:22:01.975 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:22:03.084 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:22:03.086 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:22:03.086 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453723 2023-09-23T07:22:03.086 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453723 -le 1695456865 ]] 2023-09-23T07:22:03.086 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:22:03.087 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\0\7\2\5\0\6\2\7\1 ]] 2023-09-23T07:22:03.087 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/3372138304 2023-09-23T07:22:05.380 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/3372138304 until 2023-09-23T08:22:04.657604+0000 (3600 sec) 2023-09-23T07:22:05.396 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:22:05.396 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/3372138304 2023-09-23T07:22:06.286 INFO:tasks.workunit.client.0.smithi140.stderr:listed 8 entries 2023-09-23T07:22:06.297 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/3372138304 2023-09-23T07:22:06.297 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:22:16.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:22:16.316 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:22:16.316 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:22:16.327 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:22:17.447 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:22:17.448 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:22:17.449 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453737 2023-09-23T07:22:17.449 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453737 -le 1695456865 ]] 2023-09-23T07:22:17.449 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:22:17.449 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:22:17.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:22:27.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:22:27.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:22:27.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:22:27.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:22:28.472 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:22:28.472 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:22:28.473 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453748 2023-09-23T07:22:28.473 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453748 -le 1695456865 ]] 2023-09-23T07:22:28.473 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:22:28.474 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:22:28.474 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:22:30.700 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:22:30.698+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:22:30.700 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:22:30.698+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:22:38.478 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:22:38.478 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:22:38.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:22:38.493 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:22:39.511 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:22:39.511 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:22:39.512 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453759 2023-09-23T07:22:39.512 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453759 -le 1695456865 ]] 2023-09-23T07:22:39.512 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:22:39.512 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:22:39.512 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:22:49.517 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:22:49.518 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:22:49.520 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:22:49.521 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:22:50.604 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:22:50.606 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:22:50.607 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453770 2023-09-23T07:22:50.607 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453770 -le 1695456865 ]] 2023-09-23T07:22:50.607 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:22:50.607 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:22:50.607 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.006+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.006+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.015+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.015+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb2c8480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.028 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.028 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba56fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.028 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.043+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.043+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.043+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba530100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.043+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dc280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3baf3ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dd100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dd000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.054+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1ddb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.055+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.055+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.055+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.055+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba531680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.056+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1ddc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.056+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dc080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.059+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.060+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.070+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.070+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.070+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb54eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.070+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb54ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb54ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb54f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.081 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:23:00.079+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb54f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:23:00.620 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:23:00.621 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:00.621 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:00.624 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:01.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:23:01.691 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:23:01.693 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453781 2023-09-23T07:23:01.693 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453781 -le 1695456865 ]] 2023-09-23T07:23:01.693 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:23:01.693 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:23:01.693 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:11.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:11.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:11.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:23:11.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:12.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3372138304 2023-09-23T07:23:12.854 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:23:12.872 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453792 2023-09-23T07:23:12.872 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453792 -le 1695456865 ]] 2023-09-23T07:23:12.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3372138304 ]] 2023-09-23T07:23:12.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3372138304 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:23:12.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:22.881 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:22.881 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:22.881 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:22.885 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:23:23.955 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:23:23.955 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:23:23.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453803 2023-09-23T07:23:23.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453803 -le 1695456865 ]] 2023-09-23T07:23:23.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:23:23.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\3\7\2\1\3\8\3\0\4 ]] 2023-09-23T07:23:23.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/3910729918 2023-09-23T07:23:26.475 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/3910729918 until 2023-09-23T08:23:25.708842+0000 (3600 sec) 2023-09-23T07:23:26.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:23:26.481 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/3910729918 2023-09-23T07:23:27.445 INFO:tasks.workunit.client.0.smithi140.stderr:listed 9 entries 2023-09-23T07:23:27.453 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/3910729918 2023-09-23T07:23:27.453 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:37.464 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:37.464 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:23:37.464 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:37.464 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:38.554 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:23:38.556 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:23:38.558 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453818 2023-09-23T07:23:38.558 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453818 -le 1695456865 ]] 2023-09-23T07:23:38.558 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:23:38.558 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:23:38.558 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:48.560 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:48.562 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:23:48.563 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:48.569 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:49.655 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:23:49.662 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:23:49.663 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453829 2023-09-23T07:23:49.663 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453829 -le 1695456865 ]] 2023-09-23T07:23:49.664 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:23:49.664 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:23:49.664 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:23:59.669 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:23:59.670 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:23:59.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:23:59.680 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb590300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.010+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.013+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.013+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.019+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.019+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.019+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.019+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb590b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.026+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.026+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.039+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.039+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.053 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.053 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb590d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.051+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb591f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.063+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.063+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.063+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.063+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.063+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.064+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.064+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:24:00.064+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:24:00.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:24:00.740 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:00.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453840 2023-09-23T07:24:00.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453840 -le 1695456865 ]] 2023-09-23T07:24:00.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:24:00.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:24:00.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:24:10.743 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:24:10.744 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:24:10.744 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:24:10.749 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:11.126 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:24:11.125+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:24:11.126 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:24:11.125+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:24:11.786 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:24:11.787 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:11.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453851 2023-09-23T07:24:11.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453851 -le 1695456865 ]] 2023-09-23T07:24:11.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:24:11.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:24:11.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:24:21.805 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:24:21.806 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:21.806 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:24:21.814 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:24:23.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/3910729918 2023-09-23T07:24:23.227 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:23.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453863 2023-09-23T07:24:23.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453863 -le 1695456865 ]] 2023-09-23T07:24:23.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/3910729918 ]] 2023-09-23T07:24:23.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/3910729918 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:24:23.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:24:33.239 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:24:33.240 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:24:33.240 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:33.250 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:24:34.404 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:24:34.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:34.405 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453874 2023-09-23T07:24:34.405 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453874 -le 1695456865 ]] 2023-09-23T07:24:34.405 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:24:34.405 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\3\9\1\0\7\2\9\9\1\8 ]] 2023-09-23T07:24:34.405 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/2709018820 2023-09-23T07:24:36.560 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/2709018820 until 2023-09-23T08:24:35.797396+0000 (3600 sec) 2023-09-23T07:24:36.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/2709018820 2023-09-23T07:24:36.574 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:24:37.481 INFO:tasks.workunit.client.0.smithi140.stderr:listed 10 entries 2023-09-23T07:24:37.490 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/2709018820 2023-09-23T07:24:37.490 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:24:47.495 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:24:47.495 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:24:47.495 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:24:47.495 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:48.533 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:24:48.534 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:48.534 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453888 2023-09-23T07:24:48.534 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453888 -le 1695456865 ]] 2023-09-23T07:24:48.534 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:24:48.535 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:24:48.535 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:24:51.507 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:24:51.506+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:24:51.507 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:24:51.506+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:24:58.544 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:24:58.546 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:24:58.574 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:24:58.584 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:24:59.638 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:24:59.640 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:24:59.641 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453899 2023-09-23T07:24:59.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453899 -le 1695456865 ]] 2023-09-23T07:24:59.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:24:59.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:24:59.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.011+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.011+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1d7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3e880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.014+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.018+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.018+0000 7f5568565700 -1 librbd::ImageState: 0x55a3baf3f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dd900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.026+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1dcc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.041+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.041+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.041+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1dcf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb2c8580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb2c9380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.042+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dcb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb8980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb9f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.058 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c8b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb09c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb09d980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba93e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.070+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb54f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.072+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:00.078 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:25:00.072+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb54f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:25:09.647 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:25:09.647 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:25:09.648 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:25:09.652 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:25:10.692 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:25:10.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:25:10.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453910 2023-09-23T07:25:10.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453910 -le 1695456865 ]] 2023-09-23T07:25:10.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:25:10.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:25:10.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:25:14.321 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:25:14.315+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:25:14.321 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:25:14.315+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:25:20.711 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:25:20.712 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:25:20.712 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:25:20.714 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:25:21.768 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:25:21.770 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:25:21.771 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453921 2023-09-23T07:25:21.771 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453921 -le 1695456865 ]] 2023-09-23T07:25:21.771 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:25:21.771 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:25:21.771 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:25:31.773 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:25:31.773 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:25:31.774 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:25:31.784 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:25:32.875 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2709018820 2023-09-23T07:25:32.875 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:25:32.876 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453932 2023-09-23T07:25:32.876 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453932 -le 1695456865 ]] 2023-09-23T07:25:32.876 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2709018820 ]] 2023-09-23T07:25:32.877 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2709018820 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:25:32.877 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:25:42.892 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:25:42.892 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:25:42.907 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:25:42.907 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:25:43.952 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:25:43.954 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:25:43.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453943 2023-09-23T07:25:43.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453943 -le 1695456865 ]] 2023-09-23T07:25:43.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:25:43.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\0\9\0\1\8\8\2\0 ]] 2023-09-23T07:25:43.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/1093616448 2023-09-23T07:25:45.692 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/1093616448 until 2023-09-23T08:25:44.876379+0000 (3600 sec) 2023-09-23T07:25:45.711 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/1093616448 2023-09-23T07:25:45.711 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:25:46.613 INFO:tasks.workunit.client.0.smithi140.stderr:listed 11 entries 2023-09-23T07:25:46.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/1093616448 2023-09-23T07:25:46.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:25:56.636 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:25:56.636 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:25:56.649 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:25:56.649 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:25:57.700 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:25:57.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:25:57.703 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453957 2023-09-23T07:25:57.703 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453957 -le 1695456865 ]] 2023-09-23T07:25:57.703 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:25:57.703 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:25:57.703 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:26:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.013+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.013+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.013+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bac44d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.013+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba531f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.016+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.016+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.016+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba531000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.016+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba530c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.017+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.017+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba531600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.035+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.035 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.035+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.043+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5be600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb2c9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.044+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.049+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.049+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.052 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bfb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bfd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.061+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.068+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.068+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.068+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5d0180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:00.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:26:00.068+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5d0380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:26:07.710 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:26:07.710 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:26:07.719 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:26:07.723 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:26:08.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:26:08.722 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:26:08.722 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453968 2023-09-23T07:26:08.723 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453968 -le 1695456865 ]] 2023-09-23T07:26:08.723 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:26:08.723 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:26:08.723 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:26:18.732 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:26:18.732 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:26:18.732 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:26:18.740 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:26:19.834 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:26:19.835 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:26:19.839 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453979 2023-09-23T07:26:19.839 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453979 -le 1695456865 ]] 2023-09-23T07:26:19.839 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:26:19.839 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:26:19.839 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:26:29.846 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:26:29.846 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:26:29.869 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:26:29.869 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:26:30.935 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:26:30.937 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:26:30.937 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695453990 2023-09-23T07:26:30.937 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695453990 -le 1695456865 ]] 2023-09-23T07:26:30.937 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:26:30.938 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:26:30.938 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:26:40.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:26:40.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:26:40.940 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:26:40.940 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:26:42.093 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1093616448 2023-09-23T07:26:42.093 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:26:42.094 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454002 2023-09-23T07:26:42.094 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454002 -le 1695456865 ]] 2023-09-23T07:26:42.094 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1093616448 ]] 2023-09-23T07:26:42.094 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1093616448 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:26:42.094 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:26:43.191 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:26:43.192 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:26:43.190+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:26:52.102 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:26:52.102 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:26:52.102 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:26:52.103 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:26:53.202 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:26:53.207 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:26:53.208 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454013 2023-09-23T07:26:53.208 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454013 -le 1695456865 ]] 2023-09-23T07:26:53.208 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:26:53.208 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\0\9\3\6\1\6\4\4\8 ]] 2023-09-23T07:26:53.208 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/2763363184 2023-09-23T07:26:55.702 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/2763363184 until 2023-09-23T08:26:54.938719+0000 (3600 sec) 2023-09-23T07:26:55.712 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:26:55.712 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/2763363184 2023-09-23T07:26:56.583 INFO:tasks.workunit.client.0.smithi140.stderr:listed 12 entries 2023-09-23T07:26:56.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/2763363184 2023-09-23T07:26:56.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:27:00.004 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.002+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.004 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.002+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.006 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.005+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.006 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.005+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb591080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.009+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.009+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb591980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.015+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.015+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.015+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb591000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.016+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb09d200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.019 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.018+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.019 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.018+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb5d0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.030+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.032 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.030+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.040+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.040+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.040+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d0780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.040+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d1980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.041+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.041+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.041+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb5d1800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.041+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d1600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.043+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.043+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba1f8580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.047+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.047+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d1c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.049+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.049+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d1e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.050+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.050+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.053+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.058+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.060 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.058+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba55c400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba55c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba55c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:00.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:27:00.061+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3ba55ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:27:06.603 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:27:06.603 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:27:06.603 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:27:06.606 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:27:07.719 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:27:07.726 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:27:07.727 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454027 2023-09-23T07:27:07.727 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454027 -le 1695456865 ]] 2023-09-23T07:27:07.727 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:27:07.728 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:27:07.728 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:27:17.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:27:17.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:27:17.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:27:17.738 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:27:18.866 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:27:18.866 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:27:18.867 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454038 2023-09-23T07:27:18.867 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454038 -le 1695456865 ]] 2023-09-23T07:27:18.867 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:27:18.867 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:27:18.867 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:27:28.875 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:27:28.879 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:27:28.879 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:27:28.879 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:27:29.943 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:27:29.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:27:29.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454049 2023-09-23T07:27:29.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454049 -le 1695456865 ]] 2023-09-23T07:27:29.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:27:29.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:27:29.947 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:27:33.097 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:27:33.097+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:27:33.098 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:27:33.097+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:27:39.953 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:27:39.953 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:27:39.954 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:27:39.957 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:27:41.076 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:27:41.076 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:27:41.077 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454061 2023-09-23T07:27:41.077 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454061 -le 1695456865 ]] 2023-09-23T07:27:41.077 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:27:41.077 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:27:41.078 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:27:51.079 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:27:51.084 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:27:51.084 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:27:51.088 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:27:52.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2763363184 2023-09-23T07:27:52.308 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:27:52.308 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454072 2023-09-23T07:27:52.308 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454072 -le 1695456865 ]] 2023-09-23T07:27:52.308 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2763363184 ]] 2023-09-23T07:27:52.308 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2763363184 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:27:52.309 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:02.317 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:28:02.317 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:28:02.318 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:28:02.324 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:28:03.353 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:28:03.355 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:28:03.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454083 2023-09-23T07:28:03.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454083 -le 1695456865 ]] 2023-09-23T07:28:03.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:28:03.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\7\6\3\3\6\3\1\8\4 ]] 2023-09-23T07:28:03.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/2646156005 2023-09-23T07:28:05.400 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/2646156005 until 2023-09-23T08:28:04.583564+0000 (3600 sec) 2023-09-23T07:28:05.415 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:28:05.435 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/2646156005 2023-09-23T07:28:06.197 INFO:tasks.workunit.client.0.smithi140.stderr:listed 13 entries 2023-09-23T07:28:06.210 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/2646156005 2023-09-23T07:28:06.211 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:16.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:28:16.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:28:16.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:28:16.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:28:17.276 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:28:17.278 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:28:17.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454097 2023-09-23T07:28:17.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454097 -le 1695456865 ]] 2023-09-23T07:28:17.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:28:17.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:28:17.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:27.282 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:28:27.282 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:28:27.282 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:28:27.289 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:28:28.392 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:28:28.392 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:28:28.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454108 2023-09-23T07:28:28.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454108 -le 1695456865 ]] 2023-09-23T07:28:28.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:28:28.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:28:28.393 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:38.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:28:38.401 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:28:38.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:28:38.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:28:39.417 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:28:39.420 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:28:39.420 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454119 2023-09-23T07:28:39.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454119 -le 1695456865 ]] 2023-09-23T07:28:39.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:28:39.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:28:39.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:49.434 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:28:49.434 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:28:49.443 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:28:49.452 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:28:50.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:28:50.530 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:28:50.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454130 2023-09-23T07:28:50.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454130 -le 1695456865 ]] 2023-09-23T07:28:50.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:28:50.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:28:50.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:28:53.387 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.381+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:28:53.387 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.381+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:28:53.387 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.381+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:28:53.390 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:28:53.390 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:28:53.390 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:28:53.391 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:28:53.388+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:29:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.015+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.015+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.015+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.016 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.016+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bf300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.016+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba1f9700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.016+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba1f8c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.019+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.019+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba1f8280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.027+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.033+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.033+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb54f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.034 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.033+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3ba1f8d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.037 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb54fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.036+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb54ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.049+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.050 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.049+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.049+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15c200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.049+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15c400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.051 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.050+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.055+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.055+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.059+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.059 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.067+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.067+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.071 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:29:00.070+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:29:00.533 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:00.533 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:00.533 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:00.534 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:01.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/2646156005 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454141 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454141 -le 1695456865 ]] 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/2646156005 ]] 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/2646156005 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:29:01.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:29:11.597 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:11.597 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:11.597 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:11.617 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:12.743 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:29:12.743 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:12.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454152 2023-09-23T07:29:12.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454152 -le 1695456865 ]] 2023-09-23T07:29:12.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:29:12.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\6\4\6\1\5\6\0\0\5 ]] 2023-09-23T07:29:12.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/4167742291 2023-09-23T07:29:14.455 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/4167742291 until 2023-09-23T08:29:13.662771+0000 (3600 sec) 2023-09-23T07:29:14.465 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:29:14.466 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/4167742291 2023-09-23T07:29:15.362 INFO:tasks.workunit.client.0.smithi140.stderr:listed 14 entries 2023-09-23T07:29:15.378 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/4167742291 2023-09-23T07:29:15.378 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:29:25.380 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:25.380 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:25.380 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:25.406 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:26.417 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:29:26.420 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:26.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454166 2023-09-23T07:29:26.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454166 -le 1695456865 ]] 2023-09-23T07:29:26.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:29:26.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:29:26.421 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:29:36.433 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:36.433 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:36.434 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:36.436 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:37.594 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:29:37.599 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:37.600 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454177 2023-09-23T07:29:37.600 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454177 -le 1695456865 ]] 2023-09-23T07:29:37.600 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:29:37.600 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:29:37.600 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:29:38.244 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:29:38.245 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:29:38.245 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:29:38.245 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:29:38.240+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:29:47.609 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:47.610 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:47.610 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:47.615 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:48.634 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:29:48.640 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:48.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454188 2023-09-23T07:29:48.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454188 -le 1695456865 ]] 2023-09-23T07:29:48.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:29:48.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:29:48.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:29:58.648 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:29:58.648 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:29:58.648 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:29:58.649 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:29:59.768 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:29:59.768 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:29:59.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454199 2023-09-23T07:29:59.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454199 -le 1695456865 ]] 2023-09-23T07:29:59.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:29:59.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:29:59.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb5bf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb1d7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.011+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dd600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb54eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb1dca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.026+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb590800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb15c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.039+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.045+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.045+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.045+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.045 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.045+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.046+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.046+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb15db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.062+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.062+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.062+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb15d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.062+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb15d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb15df80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.066 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb15dd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aa180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb6aa380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:00.067 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:30:00.066+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aa580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:30:09.782 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:30:09.782 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:30:09.783 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:30:09.791 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:30:10.895 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/4167742291 2023-09-23T07:30:10.897 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:30:10.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454210 2023-09-23T07:30:10.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454210 -le 1695456865 ]] 2023-09-23T07:30:10.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/4167742291 ]] 2023-09-23T07:30:10.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/4167742291 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:30:10.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:30:20.903 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:30:20.906 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:30:20.907 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:30:20.907 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:30:21.964 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:30:21.964 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:30:21.966 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454221 2023-09-23T07:30:21.966 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454221 -le 1695456865 ]] 2023-09-23T07:30:21.966 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:30:21.966 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\6\7\7\4\2\2\9\1 ]] 2023-09-23T07:30:21.967 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/771342127 2023-09-23T07:30:23.761 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/771342127 until 2023-09-23T08:30:22.993006+0000 (3600 sec) 2023-09-23T07:30:23.779 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/771342127 2023-09-23T07:30:23.785 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:30:24.694 INFO:tasks.workunit.client.0.smithi140.stderr:listed 15 entries 2023-09-23T07:30:24.709 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/771342127 2023-09-23T07:30:24.709 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:30:34.716 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:30:34.716 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:30:34.716 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:30:34.724 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:30:35.788 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:30:35.789 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:30:35.789 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454235 2023-09-23T07:30:35.789 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454235 -le 1695456865 ]] 2023-09-23T07:30:35.790 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:30:35.790 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:30:35.790 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:30:45.794 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:30:45.794 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:30:45.795 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:30:45.823 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:30:46.908 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:30:46.912 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:30:46.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454246 2023-09-23T07:30:46.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454246 -le 1695456865 ]] 2023-09-23T07:30:46.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:30:46.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:30:46.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:30:56.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:30:56.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:30:56.925 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:30:56.938 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:30:58.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:30:58.020 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:30:58.020 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454258 2023-09-23T07:30:58.020 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454258 -le 1695456865 ]] 2023-09-23T07:30:58.020 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:30:58.021 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:30:58.021 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:31:00.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb1dd680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.009+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.019 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.018+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.019 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.018+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.019 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.018+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.020 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.018+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.019+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.019+0000 7f5568565700 -1 librbd::ImageState: 0x55a3badb9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.028+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.029 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.028+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3badb9e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.030 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.029+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.030 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.029+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.030 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.029+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5d1c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.031 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.029+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6aa480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3baf3f000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6aa580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.040+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.042+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.042+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6ab380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.045+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.046 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.045+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6ab180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.058+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.058+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6aaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.059+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6aad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6ab980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:31:00.071+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6ab800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:31:08.031 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:31:08.044 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:31:08.044 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:31:08.056 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:31:09.192 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:31:09.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:31:09.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454269 2023-09-23T07:31:09.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454269 -le 1695456865 ]] 2023-09-23T07:31:09.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:31:09.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:31:09.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:31:09.226 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:31:09.227 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:31:09.228 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:31:09.221+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:31:19.203 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:31:19.203 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:31:19.206 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:31:19.208 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:31:20.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/771342127 2023-09-23T07:31:20.314 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:31:20.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454280 2023-09-23T07:31:20.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454280 -le 1695456865 ]] 2023-09-23T07:31:20.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/771342127 ]] 2023-09-23T07:31:20.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/771342127 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:31:20.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:31:30.320 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:31:30.321 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:31:30.321 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:31:30.323 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:31:31.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:31:31.402 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:31:31.402 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454291 2023-09-23T07:31:31.402 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454291 -le 1695456865 ]] 2023-09-23T07:31:31.402 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:31:31.403 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\7\7\1\3\4\2\1\2\7 ]] 2023-09-23T07:31:31.403 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/418208292 2023-09-23T07:31:33.848 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/418208292 until 2023-09-23T08:31:33.052020+0000 (3600 sec) 2023-09-23T07:31:33.865 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:31:33.871 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/418208292 2023-09-23T07:31:34.660 INFO:tasks.workunit.client.0.smithi140.stderr:listed 16 entries 2023-09-23T07:31:34.671 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/418208292 2023-09-23T07:31:34.671 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:31:44.678 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:31:44.678 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:31:44.678 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:31:44.686 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:31:45.790 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:31:45.793 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:31:45.832 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454305 2023-09-23T07:31:45.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454305 -le 1695456865 ]] 2023-09-23T07:31:45.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:31:45.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:31:45.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:31:55.834 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:31:55.837 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:31:55.837 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:31:55.849 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:31:56.918 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:31:56.919 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:31:56.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454316 2023-09-23T07:31:56.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454316 -le 1695456865 ]] 2023-09-23T07:31:56.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:31:56.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:31:56.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.008+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.008+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bb6ab680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.009+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.009+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.009+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6abb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.017 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.009+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6abd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.023+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.023+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.023+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6abf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.023+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.025+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.025+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.052+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5567d64700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bada0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bada1380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bada1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.063 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.060+0000 7f5567d64700 -1 librbd::ImageState: 0x55a3bada1980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.061+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.064 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.061+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.065 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.064+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.065 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.064+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.069+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:00.070 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:32:00.069+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:32:06.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:32:06.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:32:06.940 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:32:06.944 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:32:08.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:32:08.054 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:32:08.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454328 2023-09-23T07:32:08.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454328 -le 1695456865 ]] 2023-09-23T07:32:08.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:32:08.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:32:08.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:32:09.891 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:32:09.891 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:32:09.891 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:32:09.891 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:32:09.891 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:32:09.892 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:32:09.890+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:32:18.057 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:32:18.058 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:32:18.058 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:32:18.059 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:32:19.192 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:32:19.192 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:32:19.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454339 2023-09-23T07:32:19.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454339 -le 1695456865 ]] 2023-09-23T07:32:19.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:32:19.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:32:19.194 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:32:29.199 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:32:29.207 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:32:29.207 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:32:29.207 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:32:30.298 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/418208292 2023-09-23T07:32:30.298 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:32:30.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454350 2023-09-23T07:32:30.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454350 -le 1695456865 ]] 2023-09-23T07:32:30.300 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/418208292 ]] 2023-09-23T07:32:30.300 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/418208292 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:32:30.300 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:32:40.303 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:32:40.304 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:32:40.304 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:32:40.312 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:32:41.502 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:32:41.502 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:32:41.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454361 2023-09-23T07:32:41.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454361 -le 1695456865 ]] 2023-09-23T07:32:41.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:32:41.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\4\1\8\2\0\8\2\9\2 ]] 2023-09-23T07:32:41.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/242711256 2023-09-23T07:32:43.868 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/242711256 until 2023-09-23T08:32:43.088754+0000 (3600 sec) 2023-09-23T07:32:43.887 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:32:43.888 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/242711256 2023-09-23T07:32:44.687 INFO:tasks.workunit.client.0.smithi140.stderr:listed 17 entries 2023-09-23T07:32:44.702 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/242711256 2023-09-23T07:32:44.702 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:32:54.708 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:32:54.709 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:32:54.709 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:32:54.716 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:32:55.852 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:32:55.852 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:32:55.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454375 2023-09-23T07:32:55.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454375 -le 1695456865 ]] 2023-09-23T07:32:55.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:32:55.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:32:55.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:33:00.018 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.013+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.018 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.013+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb15dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bf180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.026 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.025+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5be980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.027+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.027+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.027+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6ab100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.027 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.027+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6ab200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6aae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d0480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.048 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.047+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d1400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.048+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.048+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.048+0000 7f5568565700 -1 librbd::ImageState: 0x55a3ba55cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.054+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.054+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.058+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.061 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.058+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb6c2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.072 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.072 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5d0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.073 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.072+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.083 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.083+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.083 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.083+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.083 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.083+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:00.083 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:33:00.083+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:33:05.854 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:33:05.855 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:33:05.855 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:33:05.857 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:33:06.719 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:33:06.720 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:33:06.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454386 2023-09-23T07:33:06.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454386 -le 1695456865 ]] 2023-09-23T07:33:06.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:33:06.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:33:06.721 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:33:16.724 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:33:16.728 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:33:16.728 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:33:16.728 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:33:17.848 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:33:17.848 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:33:17.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454397 2023-09-23T07:33:17.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454397 -le 1695456865 ]] 2023-09-23T07:33:17.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:33:17.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:33:17.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:33:27.857 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:33:27.858 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:33:27.858 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:33:27.864 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:33:28.909 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:33:28.909 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:33:28.910 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454408 2023-09-23T07:33:28.911 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454408 -le 1695456865 ]] 2023-09-23T07:33:28.911 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:33:28.911 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:33:28.911 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:33:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:33:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:33:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:33:38.929 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:33:40.027 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/242711256 2023-09-23T07:33:40.028 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:33:40.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454420 2023-09-23T07:33:40.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454420 -le 1695456865 ]] 2023-09-23T07:33:40.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/242711256 ]] 2023-09-23T07:33:40.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/242711256 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:33:40.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:33:50.039 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:33:50.039 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:33:50.039 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:33:50.045 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:33:51.195 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:33:51.195 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:33:51.196 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454431 2023-09-23T07:33:51.196 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454431 -le 1695456865 ]] 2023-09-23T07:33:51.196 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:33:51.196 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\2\4\2\7\1\1\2\5\6 ]] 2023-09-23T07:33:51.196 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/601208279 2023-09-23T07:33:52.965 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/601208279 until 2023-09-23T08:33:52.200745+0000 (3600 sec) 2023-09-23T07:33:52.981 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/601208279 2023-09-23T07:33:52.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:33:53.906 INFO:tasks.workunit.client.0.smithi140.stderr:listed 18 entries 2023-09-23T07:33:53.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/601208279 2023-09-23T07:33:53.923 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:00.004 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.003+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.004 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.003+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.008+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.008+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6ab380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.008+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.008+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.019+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.021 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.019+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6ab480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.024+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.025 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.024+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.042+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.042+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.042+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.042+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.054 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.052+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.055 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.056 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.053+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c2c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.072 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.071+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.072 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.071+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bada1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:00.082 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:34:00.080+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb6c3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:34:03.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:03.935 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:03.935 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:34:03.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:04.984 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:34:04.984 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:34:04.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454444 2023-09-23T07:34:04.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454444 -le 1695456865 ]] 2023-09-23T07:34:04.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:34:04.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:34:04.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:14.987 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:14.987 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:15.000 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:34:15.000 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:16.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:34:16.111 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:34:16.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454456 2023-09-23T07:34:16.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454456 -le 1695456865 ]] 2023-09-23T07:34:16.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:34:16.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:34:16.113 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:26.121 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:34:26.121 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:26.127 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:26.127 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:27.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:34:27.233 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:34:27.234 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454467 2023-09-23T07:34:27.234 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454467 -le 1695456865 ]] 2023-09-23T07:34:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:34:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:34:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:37.246 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:37.246 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:34:37.246 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:37.249 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454478 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454478 -le 1695456865 ]] 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:34:38.310 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:48.313 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:48.313 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:34:48.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:48.328 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:49.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/601208279 2023-09-23T07:34:49.479 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:34:49.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454489 2023-09-23T07:34:49.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454489 -le 1695456865 ]] 2023-09-23T07:34:49.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/601208279 ]] 2023-09-23T07:34:49.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/601208279 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:34:49.480 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:34:59.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:34:59.493 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:34:59.494 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:34:59.499 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:00.295 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:35:00.296 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:35:00.297 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:35:00.297 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:35:00.297 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:35:00.294+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:35:00.599 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:00.600 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:00.601 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454500 2023-09-23T07:35:00.601 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454500 -le 1695456865 ]] 2023-09-23T07:35:00.601 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:00.601 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\6\0\1\2\0\8\2\7\9 ]] 2023-09-23T07:35:00.601 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist add 172.21.15.140:0/1974081988 2023-09-23T07:35:02.042 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.041+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3ba5ff800 image watch failed: 94161715360768, (107) Transport endpoint is not connected 2023-09-23T07:35:02.043 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.041+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba5ff800 handle_error: handle=94161715360768: (107) Transport endpoint is not connected 2023-09-23T07:35:02.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.043+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3ba8740a0 handle_unwatch client blocklisted 2023-09-23T07:35:02.044 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.043+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba5ff800 handle_rewatch: client blocklisted 2023-09-23T07:35:02.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.054+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3ba5b0000 image watch failed: 94161715359744, (107) Transport endpoint is not connected 2023-09-23T07:35:02.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.054+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba5b0000 handle_error: handle=94161715359744: (107) Transport endpoint is not connected 2023-09-23T07:35:02.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.054+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3ba5b0300 image watch failed: 94161715405824, (107) Transport endpoint is not connected 2023-09-23T07:35:02.057 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.054+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba5b0300 handle_error: handle=94161715405824: (107) Transport endpoint is not connected 2023-09-23T07:35:02.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.061+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3ba221c70 handle_unwatch client blocklisted 2023-09-23T07:35:02.062 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.061+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba5b0300 handle_rewatch: client blocklisted 2023-09-23T07:35:02.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.067+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3bb067c70 handle_unwatch client blocklisted 2023-09-23T07:35:02.068 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.067+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba5b0000 handle_rewatch: client blocklisted 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.184+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3bb168300 failed to notify quiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568d66700 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3ba806600 image watch failed: 94161715323904, (107) Transport endpoint is not connected 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba806600 handle_error: handle=94161715323904: (107) Transport endpoint is not connected 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3bb9a8000 image watch failed: 94161715305472, (107) Transport endpoint is not connected 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb9a8000 handle_error: handle=94161715305472: (107) Transport endpoint is not connected 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3bb52e000 image watch failed: 94161715358720, (107) Transport endpoint is not connected 2023-09-23T07:35:02.187 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb52e000 handle_error: handle=94161715358720: (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3bb168300 image watch failed: 94161715324928, (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb168300 handle_error: handle=94161715324928: (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3ba806000 image watch failed: 94161697229824, (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba806000 handle_error: handle=94161697229824: (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3bb52e300 image watch failed: 94161715299328, (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb52e300 handle_error: handle=94161715299328: (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::ImageWatcher: 0x55a3ba806300 image watch failed: 94161715361792, (107) Transport endpoint is not connected 2023-09-23T07:35:02.188 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.185+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba806300 handle_error: handle=94161715361792: (107) Transport endpoint is not connected 2023-09-23T07:35:02.191 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.189+0000 7f5568565700 -1 librbd::watcher::RewatchRequest: 0x55a3bad54370 handle_unwatch client blocklisted 2023-09-23T07:35:02.191 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.189+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba806600 handle_rewatch: client blocklisted 2023-09-23T07:35:02.191 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.189+0000 7f5568565700 -1 librbd::watcher::RewatchRequest: 0x55a3bac110e0 handle_unwatch client blocklisted 2023-09-23T07:35:02.191 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.189+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb9a8000 handle_rewatch: client blocklisted 2023-09-23T07:35:02.191 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3bb12de50 handle_unwatch client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3ba806000 handle_rewatch: client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3ba2204b0 handle_unwatch client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3bb52e000 handle_rewatch: client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::watcher::RewatchRequest: 0x55a3bb12d720 handle_unwatch client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::Watcher: 0x55a3bb168300 handle_rewatch: client blocklisted 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::ManagedLock: 0x55a3baa2e018 send_reacquire_lock: aborting reacquire due to invalid watch handle 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4b60 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.192 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.190+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb9b8160 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.193 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.192+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.197 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.192+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb97bce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.197 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.192+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9b6fc0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.197 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.193+0000 7f5568565700 -1 librbd::watcher::RewatchRequest: 0x55a3bb067c20 handle_unwatch client blocklisted 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.193+0000 7f5568565700 -1 librbd::Watcher: 0x55a3bb52e300 handle_rewatch: client blocklisted 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.193+0000 7f5568565700 -1 librbd::watcher::RewatchRequest: 0x55a3ba1d1680 handle_unwatch client blocklisted 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.193+0000 7f5568565700 -1 librbd::Watcher: 0x55a3ba806300 handle_rewatch: client blocklisted 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.193+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.196+0000 7f5568d66700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3ba9a4b40 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.198 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.196+0000 7f5568d66700 -1 librbd::ManagedLock: 0x55a3baa2e018 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T07:35:02.214 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.210+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.214 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.210+0000 7f5568d66700 -1 librbd::ImageState: 0x55a3bb5bf000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.215 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.214+0000 7f5568d66700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:02.216 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:02.215+0000 7f5568565700 -1 librbd::ImageState: 0x55a3bb5bf000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.011 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4d00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb9a54a0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb97bce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.009+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9b6380 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4680 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb8cfb80 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.012 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4820 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.010+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb953600 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.013 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3bb2832b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb953ce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb9b8160 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.011+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9564c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3bb00d980 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.014 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb9b8420 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9b6bc0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb9b82c0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.012+0000 7f5568565700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9a60c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.013+0000 7f5568565700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3ba946c60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.013+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.015 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.014+0000 7f5568d66700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3bb1d47e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.018 INFO:tasks.workunit.client.0.smithi140.stderr:blocklisting 172.21.15.140:0/1974081988 until 2023-09-23T08:35:02.245776+0000 (3600 sec) 2023-09-23T07:35:03.032 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.029+0000 7f5568565700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.032 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.029+0000 7f5568565700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.029+0000 7f5568565700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3baadc270 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.033 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.029+0000 7f5568565700 -1 librbd::io::AioCompletion: 0x55a3bb9a51e0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.038 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.035+0000 7f5568565700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3ba661bc0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.038 INFO:tasks.workunit.client.0.smithi140.stderr:+ ceph osd blocklist ls 2023-09-23T07:35:03.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.037+0000 7f5568d66700 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.037+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4c30 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.037+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb97ba20 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.040 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep -q 172.21.15.140:0/1974081988 2023-09-23T07:35:03.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.038+0000 7f5568d66700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.038+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb953600 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.040 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.038+0000 7f5568d66700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb99f600 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb97bce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55a3bb9b6080 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.039+0000 7f5568d66700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3ba9a4e40 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.041 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:03.040+0000 7f5568d66700 -1 librbd::managed_lock::ReleaseRequest: 0x55a3badd6c60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:03.816 INFO:tasks.workunit.client.0.smithi140.stderr:listed 19 entries 2023-09-23T07:35:03.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ PREV_CLIENT=172.21.15.140:0/1974081988 2023-09-23T07:35:03.833 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:35:11.854 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::image_watcher::NotifyLockOwner: 0x55a3ba93f980 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:11.854 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3bb9a8000 error requesting lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:11.854 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3bb9a8000 blocklisted waiting for exclusive lock 2023-09-23T07:35:11.855 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::ManagedLock: 0x55a3ba00c9d8 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T07:35:11.855 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3ba0e4a90 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:11.855 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.853+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb6e3e40 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::image_watcher::NotifyLockOwner: 0x55a3bada1680 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3ba5b0300 error requesting lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::ImageWatcher: 0x55a3ba5b0300 blocklisted waiting for exclusive lock 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::ManagedLock: 0x55a3ba00cd18 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55a3bb282680 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:12.000 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T07:35:11.997+0000 7f5568d66700 -1 librbd::io::AioCompletion: 0x55a3bb97b8c0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T07:35:13.839 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:35:13.841 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:13.842 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:35:13.844 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:35:14.899 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:14.900 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:14.900 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454514 2023-09-23T07:35:14.900 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454514 -le 1695456865 ]] 2023-09-23T07:35:14.901 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:14.901 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:35:14.901 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:35:24.915 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:35:24.915 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:24.916 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:35:24.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:35:25.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:25.920 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:25.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454525 2023-09-23T07:35:25.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454525 -le 1695456865 ]] 2023-09-23T07:35:25.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:25.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:35:25.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:35:35.928 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:35:35.928 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:35.928 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:35:35.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:35:37.175 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454537 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454537 -le 1695456865 ]] 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:35:37.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:35:47.178 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:35:47.181 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:47.181 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:35:47.190 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:35:48.325 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:48.330 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:48.331 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454548 2023-09-23T07:35:48.331 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454548 -le 1695456865 ]] 2023-09-23T07:35:48.331 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:48.331 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:35:48.331 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:35:58.058 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:35:58.056+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:35:58.059 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:35:58.056+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:35:58.341 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:35:58.350 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:35:58.350 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:35:58.350 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:35:59.499 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:35:59.499 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:35:59.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454559 2023-09-23T07:35:59.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454559 -le 1695456865 ]] 2023-09-23T07:35:59.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:35:59.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:35:59.500 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:36:09.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:36:09.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:36:09.510 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:36:09.516 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:36:10.564 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:36:10.564 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:36:10.564 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454570 2023-09-23T07:36:10.564 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454570 -le 1695456865 ]] 2023-09-23T07:36:10.564 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:36:10.565 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:36:10.565 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:36:16.519 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.515+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.515+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.515+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.515+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.517+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.517+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.517+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.517+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.519+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.519+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:36:16.520 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:36:16.519+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:36:20.578 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:36:20.578 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:36:20.578 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:36:20.579 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:36:21.689 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454581 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454581 -le 1695456865 ]] 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:36:21.691 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:36:31.697 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:36:31.697 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:36:31.702 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:36:31.703 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:36:32.704 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:36:32.713 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:36:32.714 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454592 2023-09-23T07:36:32.714 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454592 -le 1695456865 ]] 2023-09-23T07:36:32.714 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:36:32.715 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:36:32.715 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:36:42.722 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:36:42.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:36:42.734 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:36:42.735 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:36:43.890 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:36:43.896 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:36:43.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454603 2023-09-23T07:36:43.899 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454603 -le 1695456865 ]] 2023-09-23T07:36:43.899 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:36:43.899 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:36:43.899 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:36:53.904 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:36:53.946 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:36:53.947 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:36:53.947 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:36:55.061 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:36:55.061 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:36:55.062 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454615 2023-09-23T07:36:55.062 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454615 -le 1695456865 ]] 2023-09-23T07:36:55.062 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:36:55.062 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:36:55.062 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:03.795 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:37:03.790+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:37:03.795 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:37:03.790+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:37:05.066 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:37:05.066 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:37:05.073 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:37:05.094 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:37:06.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:37:06.110 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:37:06.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454626 2023-09-23T07:37:06.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454626 -le 1695456865 ]] 2023-09-23T07:37:06.110 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:37:06.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:37:06.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:16.112 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:37:16.115 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:37:16.128 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:37:16.128 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:37:17.240 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454637 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454637 -le 1695456865 ]] 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:37:17.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:27.251 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:37:27.251 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:37:27.251 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:37:27.268 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:37:28.548 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454648 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454648 -le 1695456865 ]] 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:37:28.560 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:32.881 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:37:32.873+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:37:32.881 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:37:32.873+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:37:38.566 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:37:38.567 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:37:38.568 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:37:38.571 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:37:39.973 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:37:39.975 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:37:39.976 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454659 2023-09-23T07:37:39.976 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454659 -le 1695456865 ]] 2023-09-23T07:37:39.976 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:37:39.976 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:37:39.976 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:49.978 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:37:49.978 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:37:49.981 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:37:49.982 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454671 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454671 -le 1695456865 ]] 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:37:51.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:37:58.571 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:37:58.572 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:37:58.573 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:37:58.573 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:37:58.573 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:37:58.564+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:38:01.132 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:01.132 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:01.132 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:01.153 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:02.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:02.361 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:02.362 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454682 2023-09-23T07:38:02.362 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454682 -le 1695456865 ]] 2023-09-23T07:38:02.362 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:02.362 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:02.362 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:38:12.367 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:12.367 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:12.367 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:12.367 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:13.438 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:13.439 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:13.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454693 2023-09-23T07:38:13.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454693 -le 1695456865 ]] 2023-09-23T07:38:13.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:13.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:13.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:38:23.445 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:23.445 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:23.448 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:23.448 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:24.501 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:24.501 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:24.502 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454704 2023-09-23T07:38:24.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454704 -le 1695456865 ]] 2023-09-23T07:38:24.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:24.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:24.503 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:38:34.505 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:34.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:34.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:34.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:35.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:35.463 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:35.465 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454715 2023-09-23T07:38:35.465 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454715 -le 1695456865 ]] 2023-09-23T07:38:35.466 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:35.466 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:35.466 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:38:45.471 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:45.471 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:45.488 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:45.499 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:46.580 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:46.589 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:46.592 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454726 2023-09-23T07:38:46.592 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454726 -le 1695456865 ]] 2023-09-23T07:38:46.592 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:46.592 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:46.592 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:38:56.598 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:38:56.598 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:38:56.603 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:38:56.610 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:38:57.694 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454737 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454737 -le 1695456865 ]] 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:38:57.697 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:39:07.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:39:07.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:39:07.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:39:07.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:39:08.845 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:39:08.845 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:39:08.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454748 2023-09-23T07:39:08.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454748 -le 1695456865 ]] 2023-09-23T07:39:08.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:39:08.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:39:08.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:39:18.854 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:39:18.855 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:39:18.855 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:39:18.855 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:39:20.008 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:39:20.008 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:39:20.009 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454760 2023-09-23T07:39:20.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454760 -le 1695456865 ]] 2023-09-23T07:39:20.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:39:20.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:39:20.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:39:30.017 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:39:30.018 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:39:30.018 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:39:30.030 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:39:31.033 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:39:31.041 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:39:31.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454771 2023-09-23T07:39:31.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454771 -le 1695456865 ]] 2023-09-23T07:39:31.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:39:31.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:39:31.051 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:39:39.931 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:39:39.938 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.930+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:39:39.955 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.950+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:39:39.955 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.950+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:39:39.955 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.950+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:39:39.955 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.950+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:39:39.955 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:39:39.950+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:39:41.047 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:39:41.047 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:39:41.056 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:39:41.056 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:39:42.143 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:39:42.143 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:39:42.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454782 2023-09-23T07:39:42.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454782 -le 1695456865 ]] 2023-09-23T07:39:42.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:39:42.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:39:42.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:39:52.148 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:39:52.148 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:39:52.148 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:39:52.153 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:39:53.085 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:39:53.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:39:53.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454793 2023-09-23T07:39:53.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454793 -le 1695456865 ]] 2023-09-23T07:39:53.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:39:53.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:39:53.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:03.097 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:03.097 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:03.097 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:03.102 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:04.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:04.170 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:04.171 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454804 2023-09-23T07:40:04.171 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454804 -le 1695456865 ]] 2023-09-23T07:40:04.171 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:04.172 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:04.172 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:14.181 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:14.188 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:14.188 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:14.189 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:15.277 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:15.277 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:15.278 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454815 2023-09-23T07:40:15.278 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454815 -le 1695456865 ]] 2023-09-23T07:40:15.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:15.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:15.279 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:25.280 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:25.280 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:25.280 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:25.281 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:26.403 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:26.408 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:26.410 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454826 2023-09-23T07:40:26.410 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454826 -le 1695456865 ]] 2023-09-23T07:40:26.410 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:26.411 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:26.411 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:36.421 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:36.422 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:36.427 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:36.428 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:40:36.760 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:40:36.761 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:40:36.761 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:40:36.761 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:40:36.761 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:40:36.761 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:40:36.758+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:40:37.542 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:37.555 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:37.557 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454837 2023-09-23T07:40:37.557 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454837 -le 1695456865 ]] 2023-09-23T07:40:37.557 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:37.557 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:37.557 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:47.566 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:47.572 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:47.572 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:47.582 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:48.586 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454848 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454848 -le 1695456865 ]] 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:48.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:40:58.589 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:40:58.589 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:40:58.591 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:40:58.595 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:40:59.648 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:40:59.648 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:40:59.649 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454859 2023-09-23T07:40:59.649 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454859 -le 1695456865 ]] 2023-09-23T07:40:59.649 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:40:59.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:40:59.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:41:09.651 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:41:09.654 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:41:09.655 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:41:09.655 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:41:10.731 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:41:10.731 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:41:10.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454870 2023-09-23T07:41:10.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454870 -le 1695456865 ]] 2023-09-23T07:41:10.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:41:10.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:41:10.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:41:20.739 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:41:20.741 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:41:20.741 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:41:20.755 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:41:21.911 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:41:21.911 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:41:21.912 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454881 2023-09-23T07:41:21.912 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454881 -le 1695456865 ]] 2023-09-23T07:41:21.912 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:41:21.912 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:41:21.912 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:41:31.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:41:31.974 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:41:31.974 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:41:31.975 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:41:33.069 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:41:33.095 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:41:33.096 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454893 2023-09-23T07:41:33.097 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454893 -le 1695456865 ]] 2023-09-23T07:41:33.097 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:41:33.097 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:41:33.097 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:41:43.072 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:41:43.072 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:41:43.091 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:41:43.105 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:41:44.219 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:41:44.219 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:41:44.220 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454904 2023-09-23T07:41:44.220 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454904 -le 1695456865 ]] 2023-09-23T07:41:44.220 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:41:44.221 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:41:44.221 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:41:54.222 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:41:54.231 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:41:54.231 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:41:54.231 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:41:55.322 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:41:55.324 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:41:55.325 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454915 2023-09-23T07:41:55.325 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454915 -le 1695456865 ]] 2023-09-23T07:41:55.325 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:41:55.326 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:41:55.326 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:42:05.333 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:42:05.333 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:42:05.333 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:42:05.335 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:42:06.427 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454926 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454926 -le 1695456865 ]] 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:42:06.428 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:42:16.437 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:42:16.437 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:42:16.440 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:42:16.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:42:17.203 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:42:17.204 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:42:17.205 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:42:17.205 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:42:17.205 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:42:17.205 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:42:17.202+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:42:17.542 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:42:17.542 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:42:17.543 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454937 2023-09-23T07:42:17.543 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454937 -le 1695456865 ]] 2023-09-23T07:42:17.543 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:42:17.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:42:17.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:42:27.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:42:27.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:42:27.566 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:42:27.571 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:42:28.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:42:28.811 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:42:28.811 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454948 2023-09-23T07:42:28.811 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454948 -le 1695456865 ]] 2023-09-23T07:42:28.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:42:28.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:42:28.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:42:38.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:42:38.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:42:38.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:42:38.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:42:39.927 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454959 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454959 -le 1695456865 ]] 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:42:39.928 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:42:49.936 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:42:49.936 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:42:49.936 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:42:49.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:42:51.123 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454971 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454971 -le 1695456865 ]] 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:42:51.131 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:01.130 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:01.130 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:01.131 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:01.131 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:02.347 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:02.354 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:02.355 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454982 2023-09-23T07:43:02.355 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454982 -le 1695456865 ]] 2023-09-23T07:43:02.355 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:02.355 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:02.355 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:12.368 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:12.368 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:12.371 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:12.383 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:13.366 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695454993 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695454993 -le 1695456865 ]] 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:13.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:23.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:23.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:23.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:23.423 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:24.587 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:24.588 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455004 2023-09-23T07:43:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455004 -le 1695456865 ]] 2023-09-23T07:43:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:34.592 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:34.592 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:34.593 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:34.608 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:35.731 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:35.736 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:35.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455015 2023-09-23T07:43:35.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455015 -le 1695456865 ]] 2023-09-23T07:43:35.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:35.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:35.737 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:45.750 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:45.751 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:45.763 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:45.764 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:46.917 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:46.917 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:46.918 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455026 2023-09-23T07:43:46.918 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455026 -le 1695456865 ]] 2023-09-23T07:43:46.918 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:46.919 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:46.919 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:43:56.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:43:56.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:43:56.921 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:43:56.941 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:43:58.003 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:43:58.003 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:43:58.003 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455038 2023-09-23T07:43:58.004 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455038 -le 1695456865 ]] 2023-09-23T07:43:58.004 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:43:58.004 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:43:58.004 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:44:08.016 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:44:08.016 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:44:08.016 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:44:08.042 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:44:09.132 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:44:09.133 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:44:09.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455049 2023-09-23T07:44:09.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455049 -le 1695456865 ]] 2023-09-23T07:44:09.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:44:09.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:44:09.134 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:44:19.143 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:44:19.143 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:44:19.143 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:44:19.146 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:44:20.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:44:20.187 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:44:20.188 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455060 2023-09-23T07:44:20.188 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455060 -le 1695456865 ]] 2023-09-23T07:44:20.189 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:44:20.189 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:44:20.189 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:44:30.204 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:44:30.204 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:44:30.204 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:44:30.204 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:44:31.303 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:44:31.303 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:44:31.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455071 2023-09-23T07:44:31.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455071 -le 1695456865 ]] 2023-09-23T07:44:31.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:44:31.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:44:31.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:44:41.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:44:41.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:44:41.309 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:44:41.309 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:44:42.462 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:44:42.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:44:42.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455082 2023-09-23T07:44:42.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455082 -le 1695456865 ]] 2023-09-23T07:44:42.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:44:42.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:44:42.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:44:52.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:44:52.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:44:52.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:44:52.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:44:53.543 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:44:53.543 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:44:53.543 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455093 2023-09-23T07:44:53.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455093 -le 1695456865 ]] 2023-09-23T07:44:53.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:44:53.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:44:53.544 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:03.553 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:03.553 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:03.553 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:03.553 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:45:04.655 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:45:04.659 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:45:04.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455104 2023-09-23T07:45:04.660 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455104 -le 1695456865 ]] 2023-09-23T07:45:04.660 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:45:04.660 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:45:04.660 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:14.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:14.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:14.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:45:14.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:15.769 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:45:15.776 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:45:15.777 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455115 2023-09-23T07:45:15.777 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455115 -le 1695456865 ]] 2023-09-23T07:45:15.777 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:45:15.777 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:45:15.777 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:45:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:26.915 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:45:26.915 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:45:26.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455126 2023-09-23T07:45:26.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455126 -le 1695456865 ]] 2023-09-23T07:45:26.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:45:26.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:45:26.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:36.922 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:45:36.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:36.931 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:36.936 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:37.942 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:45:37.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:45:37.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455137 2023-09-23T07:45:37.950 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455137 -le 1695456865 ]] 2023-09-23T07:45:37.951 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:45:37.951 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:45:37.951 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:47.953 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:47.954 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:47.980 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:45:47.980 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:49.071 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:45:49.072 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:45:49.072 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455149 2023-09-23T07:45:49.073 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455149 -le 1695456865 ]] 2023-09-23T07:45:49.073 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:45:49.073 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:45:49.073 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:45:59.075 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:45:59.075 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:45:59.075 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:45:59.075 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:00.259 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:00.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:00.260 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455160 2023-09-23T07:46:00.261 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455160 -le 1695456865 ]] 2023-09-23T07:46:00.261 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:00.261 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:00.261 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:46:10.278 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:46:10.278 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:10.278 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:46:10.299 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:46:11.357 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:11.357 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:11.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455171 2023-09-23T07:46:11.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455171 -le 1695456865 ]] 2023-09-23T07:46:11.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:11.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:11.358 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:46:21.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:46:21.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:46:21.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:46:21.367 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:22.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:22.552 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:22.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455182 2023-09-23T07:46:22.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455182 -le 1695456865 ]] 2023-09-23T07:46:22.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:22.554 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:22.554 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:46:32.559 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:46:32.560 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:46:32.560 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:32.563 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:46:32.860 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:46:32.849+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:46:32.860 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:46:32.849+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:46:33.634 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:33.635 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:33.637 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455193 2023-09-23T07:46:33.637 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455193 -le 1695456865 ]] 2023-09-23T07:46:33.637 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:33.637 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:33.637 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:46:43.643 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:46:43.643 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:46:43.644 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:46:43.644 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:44.750 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:44.760 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:44.760 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455204 2023-09-23T07:46:44.760 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455204 -le 1695456865 ]] 2023-09-23T07:46:44.761 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:44.761 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:44.761 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:46:54.762 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:46:54.767 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:46:54.768 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:46:54.768 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:46:55.918 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:46:55.919 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:46:55.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455215 2023-09-23T07:46:55.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455215 -le 1695456865 ]] 2023-09-23T07:46:55.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:46:55.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:46:55.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:47:05.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:47:05.923 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:47:05.927 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:47:05.941 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:47:07.079 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:47:07.079 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:47:07.080 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455227 2023-09-23T07:47:07.080 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455227 -le 1695456865 ]] 2023-09-23T07:47:07.081 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:47:07.081 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:47:07.081 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:47:17.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:47:17.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:47:17.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:47:17.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455238 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455238 -le 1695456865 ]] 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:47:18.193 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:47:25.498 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:47:25.487+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:47:25.498 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:47:25.487+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:47:28.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:47:28.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:47:28.194 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:47:28.210 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:47:29.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:47:29.318 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:47:29.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455249 2023-09-23T07:47:29.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455249 -le 1695456865 ]] 2023-09-23T07:47:29.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:47:29.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:47:29.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:47:39.321 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:47:39.339 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:47:39.339 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:47:39.339 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:47:40.514 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:47:40.515 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:47:40.515 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455260 2023-09-23T07:47:40.516 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455260 -le 1695456865 ]] 2023-09-23T07:47:40.516 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:47:40.516 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:47:40.516 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:47:50.517 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:47:50.517 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:47:50.526 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:47:50.527 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:47:51.677 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:47:51.677 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:47:51.678 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455271 2023-09-23T07:47:51.678 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455271 -le 1695456865 ]] 2023-09-23T07:47:51.678 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:47:51.678 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:47:51.678 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:01.686 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:01.687 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:01.687 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:01.687 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:02.944 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:02.944 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:02.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455282 2023-09-23T07:48:02.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455282 -le 1695456865 ]] 2023-09-23T07:48:02.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:02.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:02.946 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:12.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:12.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:12.958 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:12.970 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:14.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:14.013 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:14.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455294 2023-09-23T07:48:14.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455294 -le 1695456865 ]] 2023-09-23T07:48:14.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:14.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:14.019 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:24.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:24.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:24.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:24.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:25.080 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:25.089 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:25.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455305 2023-09-23T07:48:25.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455305 -le 1695456865 ]] 2023-09-23T07:48:25.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:25.090 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:25.091 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:35.094 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:35.095 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:35.114 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:35.115 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:36.226 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:36.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:36.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455316 2023-09-23T07:48:36.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455316 -le 1695456865 ]] 2023-09-23T07:48:36.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:36.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:36.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:46.230 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:46.230 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:46.230 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:46.241 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:47.303 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:47.303 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:47.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455327 2023-09-23T07:48:47.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455327 -le 1695456865 ]] 2023-09-23T07:48:47.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:47.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:47.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:48:57.328 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:48:57.328 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:48:57.328 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:48:57.328 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:48:58.467 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:48:58.468 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:48:58.468 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455338 2023-09-23T07:48:58.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455338 -le 1695456865 ]] 2023-09-23T07:48:58.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:48:58.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:48:58.469 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:49:08.471 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:49:08.471 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:49:08.471 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:49:08.489 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:49:09.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:49:09.626 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:49:09.627 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455349 2023-09-23T07:49:09.627 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455349 -le 1695456865 ]] 2023-09-23T07:49:09.628 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:49:09.628 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:49:09.628 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:49:19.634 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:49:19.634 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:49:19.636 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:49:19.637 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:49:20.811 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:49:20.811 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:49:20.811 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455360 2023-09-23T07:49:20.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455360 -le 1695456865 ]] 2023-09-23T07:49:20.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:49:20.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:49:20.812 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:49:30.823 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:49:30.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:49:30.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:49:30.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:49:31.803 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455371 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455371 -le 1695456865 ]] 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:49:31.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:49:41.808 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:49:41.818 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:49:41.819 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:49:41.836 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:49:42.885 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:49:42.886 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:49:42.888 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455382 2023-09-23T07:49:42.888 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455382 -le 1695456865 ]] 2023-09-23T07:49:42.888 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:49:42.889 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:49:42.889 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:49:52.896 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:49:52.896 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:49:52.896 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:49:52.912 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:49:54.106 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:49:54.110 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:49:54.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455394 2023-09-23T07:49:54.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455394 -le 1695456865 ]] 2023-09-23T07:49:54.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:49:54.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:49:54.111 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:04.120 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:04.121 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:04.124 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:50:04.128 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:05.184 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455405 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455405 -le 1695456865 ]] 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:50:05.186 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:15.195 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:50:15.195 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:15.209 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:15.209 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:16.357 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455416 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455416 -le 1695456865 ]] 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:50:16.360 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:26.368 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:26.368 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:50:26.368 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:26.374 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:27.457 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:50:27.458 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:50:27.458 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455427 2023-09-23T07:50:27.458 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455427 -le 1695456865 ]] 2023-09-23T07:50:27.458 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:50:27.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:50:27.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:37.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:37.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:50:37.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:37.475 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:38.614 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:50:38.614 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:50:38.614 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455438 2023-09-23T07:50:38.615 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455438 -le 1695456865 ]] 2023-09-23T07:50:38.615 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:50:38.615 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:50:38.615 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:48.626 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:50:48.627 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:48.627 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:48.627 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:49.783 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:50:49.783 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:50:49.784 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455449 2023-09-23T07:50:49.785 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455449 -le 1695456865 ]] 2023-09-23T07:50:49.785 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:50:49.785 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:50:49.785 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:50:55.265 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:50:55.257+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T07:50:55.265 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T07:50:55.257+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T07:50:59.794 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:50:59.813 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:50:59.813 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:50:59.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:00.942 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:00.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:00.943 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455460 2023-09-23T07:51:00.943 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455460 -le 1695456865 ]] 2023-09-23T07:51:00.944 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:00.944 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:00.944 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:51:10.945 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:51:10.950 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:51:10.951 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:10.977 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:51:12.141 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:12.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455472 2023-09-23T07:51:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455472 -le 1695456865 ]] 2023-09-23T07:51:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:51:22.163 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:22.164 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:51:22.171 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:51:22.171 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:51:23.288 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:23.288 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:23.289 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455483 2023-09-23T07:51:23.289 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455483 -le 1695456865 ]] 2023-09-23T07:51:23.289 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:23.289 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:23.289 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:51:33.294 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:51:33.296 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:51:33.309 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:33.311 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:51:34.449 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:34.449 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:34.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455494 2023-09-23T07:51:34.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455494 -le 1695456865 ]] 2023-09-23T07:51:34.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:34.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:34.450 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:51:44.457 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:44.457 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:51:44.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:51:44.479 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:51:45.621 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455505 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455505 -le 1695456865 ]] 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:45.624 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:51:55.634 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:51:55.640 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:51:55.641 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:51:55.651 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:51:56.743 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:51:56.743 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:51:56.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455516 2023-09-23T07:51:56.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455516 -le 1695456865 ]] 2023-09-23T07:51:56.744 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:51:56.745 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:51:56.745 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:52:06.751 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:52:06.752 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:52:06.752 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:52:06.761 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:52:07.813 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:52:07.814 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:52:07.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455527 2023-09-23T07:52:07.814 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455527 -le 1695456865 ]] 2023-09-23T07:52:07.815 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:52:07.815 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:52:07.815 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:52:17.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:52:17.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:52:17.826 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:52:17.826 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:52:18.861 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:52:18.861 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:52:18.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455538 2023-09-23T07:52:18.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455538 -le 1695456865 ]] 2023-09-23T07:52:18.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:52:18.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:52:18.862 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:52:28.891 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:52:28.891 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:52:28.891 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:52:28.900 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:52:29.953 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:52:29.953 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:52:29.960 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455549 2023-09-23T07:52:29.961 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455549 -le 1695456865 ]] 2023-09-23T07:52:29.961 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:52:29.961 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:52:29.961 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:52:39.964 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:52:39.964 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:52:39.964 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:52:39.964 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:52:41.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:52:41.059 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:52:41.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455561 2023-09-23T07:52:41.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455561 -le 1695456865 ]] 2023-09-23T07:52:41.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:52:41.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:52:41.060 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:52:51.067 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:52:51.067 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:52:51.067 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:52:51.071 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:52:52.117 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:52:52.117 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:52:52.117 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455572 2023-09-23T07:52:52.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455572 -le 1695456865 ]] 2023-09-23T07:52:52.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:52:52.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:52:52.118 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:02.117 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:02.117 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:02.125 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:02.162 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:03.295 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:03.297 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:03.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455583 2023-09-23T07:53:03.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455583 -le 1695456865 ]] 2023-09-23T07:53:03.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:03.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:03.299 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:13.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:13.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:13.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:13.310 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:14.377 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:14.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:14.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455594 2023-09-23T07:53:14.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455594 -le 1695456865 ]] 2023-09-23T07:53:14.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:14.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:14.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:24.399 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:24.399 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:24.399 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:24.399 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:25.519 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:25.519 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:25.520 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455605 2023-09-23T07:53:25.520 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455605 -le 1695456865 ]] 2023-09-23T07:53:25.520 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:25.520 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:25.520 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:35.525 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:35.525 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:35.526 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:35.527 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:36.633 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:36.634 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:36.634 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455616 2023-09-23T07:53:36.634 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455616 -le 1695456865 ]] 2023-09-23T07:53:36.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:36.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:36.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:46.636 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:46.637 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:46.660 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:46.667 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:47.734 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:47.740 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:47.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455627 2023-09-23T07:53:47.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455627 -le 1695456865 ]] 2023-09-23T07:53:47.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:47.741 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:47.742 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:53:57.748 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:53:57.748 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:53:57.752 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:53:57.752 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:53:58.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:53:58.924 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:53:58.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455638 2023-09-23T07:53:58.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455638 -le 1695456865 ]] 2023-09-23T07:53:58.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:53:58.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:53:58.925 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:54:08.931 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:54:08.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:54:08.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:54:08.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:54:10.026 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:54:10.027 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:54:10.027 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455650 2023-09-23T07:54:10.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455650 -le 1695456865 ]] 2023-09-23T07:54:10.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:54:10.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:54:10.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:54:20.031 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:54:20.031 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:54:20.031 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:54:20.039 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:54:21.175 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:54:21.175 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:54:21.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455661 2023-09-23T07:54:21.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455661 -le 1695456865 ]] 2023-09-23T07:54:21.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:54:21.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:54:21.176 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:54:31.184 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:54:31.184 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:54:31.190 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:54:31.190 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:54:32.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:54:32.228 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:54:32.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455672 2023-09-23T07:54:32.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455672 -le 1695456865 ]] 2023-09-23T07:54:32.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:54:32.228 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:54:32.229 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:54:42.244 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:54:42.244 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:54:42.244 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:54:42.256 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:54:43.345 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:54:43.345 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:54:43.346 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455683 2023-09-23T07:54:43.346 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455683 -le 1695456865 ]] 2023-09-23T07:54:43.346 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:54:43.346 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:54:43.347 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:54:53.348 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:54:53.348 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:54:53.366 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:54:53.366 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:54:54.440 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:54:54.440 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:54:54.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455694 2023-09-23T07:54:54.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455694 -le 1695456865 ]] 2023-09-23T07:54:54.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:54:54.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:54:54.441 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:55:04.453 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:55:04.454 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:55:04.454 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:55:04.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:55:05.546 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:55:05.546 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:55:05.547 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455705 2023-09-23T07:55:05.547 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455705 -le 1695456865 ]] 2023-09-23T07:55:05.547 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:55:05.547 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:55:05.547 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:55:15.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:55:15.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:55:15.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:55:15.558 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:55:16.586 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:55:16.588 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:55:16.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455716 2023-09-23T07:55:16.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455716 -le 1695456865 ]] 2023-09-23T07:55:16.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:55:16.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:55:16.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:55:26.593 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:55:26.593 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:55:26.594 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:55:26.602 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:55:27.727 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:55:27.730 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:55:27.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455727 2023-09-23T07:55:27.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455727 -le 1695456865 ]] 2023-09-23T07:55:27.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:55:27.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:55:27.732 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:55:37.742 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:55:37.742 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:55:37.742 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:55:37.745 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:55:38.919 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:55:38.920 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:55:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455738 2023-09-23T07:55:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455738 -le 1695456865 ]] 2023-09-23T07:55:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:55:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:55:38.921 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.669+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:55:48.673 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:55:48.674 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:55:48.670+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:55:48.929 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:55:48.930 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:55:48.930 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:55:48.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:55:50.055 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455750 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455750 -le 1695456865 ]] 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:55:50.056 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:00.053 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:00.053 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:00.054 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:00.061 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:01.223 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:01.224 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:01.225 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455761 2023-09-23T07:56:01.225 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455761 -le 1695456865 ]] 2023-09-23T07:56:01.225 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:01.225 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:01.225 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:11.234 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:11.234 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:11.241 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:11.250 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:11.929 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T07:56:11.929 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T07:56:11.929 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T07:56:11.930 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T07:56:11.931 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T07:56:11.927+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T07:56:12.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:12.315 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:12.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455772 2023-09-23T07:56:12.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455772 -le 1695456865 ]] 2023-09-23T07:56:12.318 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:12.318 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:12.318 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:22.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:22.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:22.324 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:22.337 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:23.461 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:23.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:23.462 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455783 2023-09-23T07:56:23.462 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455783 -le 1695456865 ]] 2023-09-23T07:56:23.462 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:23.462 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:23.463 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:33.467 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:33.468 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:33.468 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:33.468 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:34.596 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:34.597 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:34.598 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455794 2023-09-23T07:56:34.598 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455794 -le 1695456865 ]] 2023-09-23T07:56:34.598 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:34.598 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:34.598 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:44.609 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:44.609 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:44.609 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:44.621 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:45.677 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:45.682 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:45.683 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455805 2023-09-23T07:56:45.684 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455805 -le 1695456865 ]] 2023-09-23T07:56:45.684 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:45.684 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:45.684 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:56:55.698 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:56:55.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:56:55.699 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:56:55.718 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:56:56.804 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455816 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455816 -le 1695456865 ]] 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:56:56.823 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:57:06.830 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:57:06.830 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:57:06.838 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:57:06.851 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:57:08.021 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455828 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455828 -le 1695456865 ]] 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:57:08.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:57:18.029 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:57:18.035 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:57:18.036 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:57:18.036 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:57:19.045 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:57:19.046 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:57:19.046 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455839 2023-09-23T07:57:19.047 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455839 -le 1695456865 ]] 2023-09-23T07:57:19.047 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:57:19.047 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:57:19.047 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:57:29.050 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:57:29.051 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:57:29.051 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:57:29.056 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:57:30.125 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:57:30.132 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:57:30.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455850 2023-09-23T07:57:30.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455850 -le 1695456865 ]] 2023-09-23T07:57:30.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:57:30.133 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:57:30.134 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:57:40.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:57:40.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:57:40.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:57:40.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:57:41.269 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:57:41.272 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:57:41.272 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455861 2023-09-23T07:57:41.272 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455861 -le 1695456865 ]] 2023-09-23T07:57:41.273 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:57:41.273 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:57:41.273 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:57:51.274 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:57:51.275 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:57:51.275 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:57:51.277 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:57:52.365 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:57:52.365 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:57:52.368 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455872 2023-09-23T07:57:52.368 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455872 -le 1695456865 ]] 2023-09-23T07:57:52.368 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:57:52.368 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:57:52.368 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:02.374 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:02.374 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:02.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:02.384 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:03.476 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:03.478 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:03.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455883 2023-09-23T07:58:03.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455883 -le 1695456865 ]] 2023-09-23T07:58:03.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:03.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:03.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:13.484 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:13.484 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:13.493 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:13.493 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:14.631 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:14.632 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:14.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455894 2023-09-23T07:58:14.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455894 -le 1695456865 ]] 2023-09-23T07:58:14.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:14.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:14.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:24.640 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:24.651 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:24.651 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:24.659 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:25.784 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:25.785 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455905 2023-09-23T07:58:25.786 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455905 -le 1695456865 ]] 2023-09-23T07:58:25.787 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:25.787 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:25.787 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:35.788 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:35.788 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:35.790 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:35.803 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:36.863 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:36.863 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:36.864 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455916 2023-09-23T07:58:36.864 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455916 -le 1695456865 ]] 2023-09-23T07:58:36.864 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:36.864 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:36.864 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:46.885 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:46.885 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:46.885 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:46.901 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:47.927 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:47.927 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:47.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455927 2023-09-23T07:58:47.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455927 -le 1695456865 ]] 2023-09-23T07:58:47.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:47.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:47.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:58:57.937 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:58:57.937 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:58:57.937 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:58:57.939 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:58:58.968 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:58:58.970 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:58:58.972 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455938 2023-09-23T07:58:58.972 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455938 -le 1695456865 ]] 2023-09-23T07:58:58.972 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:58:58.972 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:58:58.972 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:59:08.975 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:59:08.983 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:59:08.983 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:59:08.990 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:59:10.141 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:59:10.141 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:59:10.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455950 2023-09-23T07:59:10.143 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455950 -le 1695456865 ]] 2023-09-23T07:59:10.143 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:59:10.143 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:59:10.143 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:59:20.151 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:59:20.152 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:59:20.152 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:59:20.152 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:59:21.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:59:21.195 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:59:21.197 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455961 2023-09-23T07:59:21.197 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455961 -le 1695456865 ]] 2023-09-23T07:59:21.197 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:59:21.197 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:59:21.197 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:59:31.203 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:59:31.215 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:59:31.215 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:59:31.215 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:59:32.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:59:32.313 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:59:32.314 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455972 2023-09-23T07:59:32.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455972 -le 1695456865 ]] 2023-09-23T07:59:32.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:59:32.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:59:32.315 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:59:42.326 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:59:42.326 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:59:42.326 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:59:42.330 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:59:43.399 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:59:43.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:59:43.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455983 2023-09-23T07:59:43.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455983 -le 1695456865 ]] 2023-09-23T07:59:43.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:59:43.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:59:43.401 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T07:59:53.415 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T07:59:53.416 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T07:59:53.416 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T07:59:53.416 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T07:59:54.531 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T07:59:54.549 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T07:59:54.549 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695455994 2023-09-23T07:59:54.549 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695455994 -le 1695456865 ]] 2023-09-23T07:59:54.549 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T07:59:54.550 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T07:59:54.550 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:00:04.538 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:00:04.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:00:04.549 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:00:04.553 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:00:05.646 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:00:05.646 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:00:05.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456005 2023-09-23T08:00:05.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456005 -le 1695456865 ]] 2023-09-23T08:00:05.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:00:05.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:00:05.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:00:15.659 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:00:15.659 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:00:15.660 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:00:15.668 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:00:16.842 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:00:16.842 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:00:16.843 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456016 2023-09-23T08:00:16.843 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456016 -le 1695456865 ]] 2023-09-23T08:00:16.843 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:00:16.843 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:00:16.843 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:00:26.848 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:00:26.848 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:00:26.848 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:00:26.851 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:00:28.050 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:00:28.050 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:00:28.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456028 2023-09-23T08:00:28.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456028 -le 1695456865 ]] 2023-09-23T08:00:28.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:00:28.119 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:00:28.120 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:00:29.048 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:00:29.047+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:00:29.049 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:00:29.047+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:00:38.124 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:00:38.124 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:00:38.125 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:00:38.130 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:00:39.250 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:00:39.251 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:00:39.252 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456039 2023-09-23T08:00:39.253 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456039 -le 1695456865 ]] 2023-09-23T08:00:39.253 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:00:39.253 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:00:39.253 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:00:49.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:00:49.260 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:00:49.261 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:00:49.269 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:00:50.456 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:00:50.457 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:00:50.458 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:00:50.458 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:00:50.458 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:00:50.446+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:00:50.458 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:00:50.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456050 2023-09-23T08:00:50.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456050 -le 1695456865 ]] 2023-09-23T08:00:50.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:00:50.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:00:50.459 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:00.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:00.461 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:00.462 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:00.472 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:01.844 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:01.844 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:01.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456061 2023-09-23T08:01:01.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456061 -le 1695456865 ]] 2023-09-23T08:01:01.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:01.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:01.846 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:11.861 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:11.862 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:11.862 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:11.862 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:12.984 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:12.984 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:12.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456072 2023-09-23T08:01:12.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456072 -le 1695456865 ]] 2023-09-23T08:01:12.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:12.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:12.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:22.991 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:22.991 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:22.992 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:22.992 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:24.416 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:24.422 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:24.590 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456084 2023-09-23T08:01:24.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456084 -le 1695456865 ]] 2023-09-23T08:01:24.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:24.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:24.591 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:34.593 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:34.594 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:34.602 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:34.615 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:35.816 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:35.817 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:35.817 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456095 2023-09-23T08:01:35.817 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456095 -le 1695456865 ]] 2023-09-23T08:01:35.817 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:35.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:35.818 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:45.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:45.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:45.827 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:45.834 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:47.179 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:47.183 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:47.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456107 2023-09-23T08:01:47.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456107 -le 1695456865 ]] 2023-09-23T08:01:47.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:47.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:47.187 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:01:57.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:01:57.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:01:57.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:01:57.201 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:01:58.378 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:01:58.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:01:58.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456118 2023-09-23T08:01:58.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456118 -le 1695456865 ]] 2023-09-23T08:01:58.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:01:58.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:01:58.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:08.394 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:02:08.394 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:02:08.394 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:02:08.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:02:09.396 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:02:09.396 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:02:09.397 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456129 2023-09-23T08:02:09.398 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456129 -le 1695456865 ]] 2023-09-23T08:02:09.398 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:02:09.398 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:02:09.398 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:19.410 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:02:19.420 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:02:19.421 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:02:19.445 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456140 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456140 -le 1695456865 ]] 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:02:20.530 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:30.535 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:02:30.536 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:02:30.541 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:02:30.544 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:02:31.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:02:31.639 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:02:31.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456151 2023-09-23T08:02:31.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456151 -le 1695456865 ]] 2023-09-23T08:02:31.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:02:31.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:02:31.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:41.642 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:02:41.642 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:02:41.642 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:02:41.646 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:02:42.658 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:02:42.658 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:02:42.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456162 2023-09-23T08:02:42.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456162 -le 1695456865 ]] 2023-09-23T08:02:42.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:02:42.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:02:42.659 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:52.671 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:02:52.671 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:02:52.691 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:02:52.691 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:02:53.764 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:02:53.764 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:02:53.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456173 2023-09-23T08:02:53.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456173 -le 1695456865 ]] 2023-09-23T08:02:53.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:02:53.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:02:53.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:02:57.592 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:02:57.589+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:03:03.770 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:03.775 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:03:03.775 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:03.775 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:04.932 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:03:04.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:03:04.933 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456184 2023-09-23T08:03:04.933 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456184 -le 1695456865 ]] 2023-09-23T08:03:04.933 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:03:04.934 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:03:04.934 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:03:14.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:14.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:03:14.943 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:14.944 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:16.188 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:03:16.190 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:03:16.191 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456196 2023-09-23T08:03:16.191 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456196 -le 1695456865 ]] 2023-09-23T08:03:16.191 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:03:16.191 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:03:16.191 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:03:26.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:26.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:03:26.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:26.194 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:27.233 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:03:27.233 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:03:27.234 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456207 2023-09-23T08:03:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456207 -le 1695456865 ]] 2023-09-23T08:03:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:03:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:03:27.235 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:03:37.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:37.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:03:37.243 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:37.244 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:37.804 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:03:37.804 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:03:37.804 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:03:37.804 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:03:37.805 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:03:37.803+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:03:38.380 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:03:38.381 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:03:38.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456218 2023-09-23T08:03:38.382 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456218 -le 1695456865 ]] 2023-09-23T08:03:38.382 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:03:38.382 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:03:38.382 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:03:48.391 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:48.392 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:03:48.395 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:48.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:49.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:03:49.390 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:03:49.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456229 2023-09-23T08:03:49.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456229 -le 1695456865 ]] 2023-09-23T08:03:49.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:03:49.391 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:03:49.392 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:03:59.394 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:03:59.403 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:03:59.403 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:03:59.403 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:00.596 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:00.596 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:00.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456240 2023-09-23T08:04:00.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456240 -le 1695456865 ]] 2023-09-23T08:04:00.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:00.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:00.597 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:04:10.603 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:04:10.606 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:10.606 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:04:10.606 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:04:11.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456251 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456251 -le 1695456865 ]] 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:11.766 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:04:21.774 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:04:21.774 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:04:21.776 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:04:21.788 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:22.865 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:22.872 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:22.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456262 2023-09-23T08:04:22.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456262 -le 1695456865 ]] 2023-09-23T08:04:22.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:22.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:22.873 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:04:23.490 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.484+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:04:23.490 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.484+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:04:23.490 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.484+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:04:23.496 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.495+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:04:23.496 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.495+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:04:23.497 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.495+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:04:23.497 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:04:23.495+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:04:32.886 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:04:32.886 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:04:32.892 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:32.893 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:04:34.026 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:34.026 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:34.026 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456274 2023-09-23T08:04:34.026 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456274 -le 1695456865 ]] 2023-09-23T08:04:34.026 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:34.027 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:34.027 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:04:44.027 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:04:44.028 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:44.028 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:04:44.028 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:04:45.173 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:45.177 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:45.177 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456285 2023-09-23T08:04:45.177 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456285 -le 1695456865 ]] 2023-09-23T08:04:45.178 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:45.178 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:45.178 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:04:55.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:04:55.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:04:55.193 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:04:55.213 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:04:56.317 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:04:56.318 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:04:56.318 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456296 2023-09-23T08:04:56.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456296 -le 1695456865 ]] 2023-09-23T08:04:56.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:04:56.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:04:56.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:06.329 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:05:06.330 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:05:06.330 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:05:06.336 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:05:07.381 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:05:07.381 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:05:07.394 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456307 2023-09-23T08:05:07.394 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456307 -le 1695456865 ]] 2023-09-23T08:05:07.394 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:05:07.394 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:05:07.395 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:17.407 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:05:17.408 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:05:17.408 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:05:17.424 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:05:18.492 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:05:18.493 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:05:18.497 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456318 2023-09-23T08:05:18.497 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456318 -le 1695456865 ]] 2023-09-23T08:05:18.497 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:05:18.497 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:05:18.497 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:28.497 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:05:28.497 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:05:28.497 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:05:28.498 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:05:29.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:05:29.635 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:05:29.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456329 2023-09-23T08:05:29.635 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456329 -le 1695456865 ]] 2023-09-23T08:05:29.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:05:29.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:05:29.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:32.877 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:05:32.874+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:05:32.877 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:05:32.874+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:05:39.217 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:05:39.218 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:05:39.219 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:05:39.219 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:05:39.219 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:05:39.219 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:05:39.219 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:05:39.214+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:05:39.643 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:05:39.658 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:05:39.671 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:05:39.671 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:05:40.774 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:05:40.774 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:05:40.774 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456340 2023-09-23T08:05:40.775 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456340 -le 1695456865 ]] 2023-09-23T08:05:40.775 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:05:40.775 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:05:40.775 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:50.778 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:05:50.778 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:05:50.779 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:05:50.783 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:05:51.841 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:05:51.849 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:05:51.849 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456351 2023-09-23T08:05:51.849 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456351 -le 1695456865 ]] 2023-09-23T08:05:51.849 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:05:51.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:05:51.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:05:53.077 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:05:53.076+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:05:53.077 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:05:53.076+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:06:01.867 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:01.867 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:01.867 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:01.884 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:03.031 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456363 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456363 -le 1695456865 ]] 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:03.032 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:06:13.036 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:13.040 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:13.040 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:13.051 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:14.177 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:14.178 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:14.178 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456374 2023-09-23T08:06:14.179 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456374 -le 1695456865 ]] 2023-09-23T08:06:14.179 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:14.179 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:14.179 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:06:24.185 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:24.185 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:24.186 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:24.186 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:25.290 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:25.292 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:25.293 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456385 2023-09-23T08:06:25.293 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456385 -le 1695456865 ]] 2023-09-23T08:06:25.293 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:25.293 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:25.293 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:06:35.294 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:35.298 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:35.303 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:35.312 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:36.378 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:36.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:36.379 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456396 2023-09-23T08:06:36.380 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456396 -le 1695456865 ]] 2023-09-23T08:06:36.380 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:36.380 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:36.380 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:06:46.390 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:46.392 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:46.400 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:46.404 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:47.534 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:47.535 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:47.537 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456407 2023-09-23T08:06:47.537 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456407 -le 1695456865 ]] 2023-09-23T08:06:47.537 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:47.537 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:47.537 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:06:57.539 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:06:57.542 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:06:57.542 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:06:57.542 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:06:58.667 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456418 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456418 -le 1695456865 ]] 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:06:58.668 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:07:08.670 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:07:08.671 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:07:08.673 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:07:08.674 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:07:09.803 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:07:09.803 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:07:09.807 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456429 2023-09-23T08:07:09.807 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456429 -le 1695456865 ]] 2023-09-23T08:07:09.807 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:07:09.807 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:07:09.807 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:07:19.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:07:19.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:07:19.824 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:07:19.838 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:07:20.914 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:07:20.914 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:07:20.915 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456440 2023-09-23T08:07:20.915 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456440 -le 1695456865 ]] 2023-09-23T08:07:20.915 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:07:20.915 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:07:20.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:07:30.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:07:30.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:07:30.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:07:30.934 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:07:32.010 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:07:32.010 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:07:32.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456452 2023-09-23T08:07:32.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456452 -le 1695456865 ]] 2023-09-23T08:07:32.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:07:32.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:07:32.011 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:07:42.020 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:07:42.026 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:07:42.026 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:07:42.035 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:07:43.158 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:07:43.159 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:07:43.159 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456463 2023-09-23T08:07:43.160 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456463 -le 1695456865 ]] 2023-09-23T08:07:43.160 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:07:43.160 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:07:43.160 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:07:53.167 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:07:53.168 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:07:53.178 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:07:53.183 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:07:54.311 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:07:54.312 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:07:54.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456474 2023-09-23T08:07:54.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456474 -le 1695456865 ]] 2023-09-23T08:07:54.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:07:54.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:07:54.313 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:04.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:08:04.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:04.322 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:04.323 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:05.335 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:08:05.335 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:08:05.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456485 2023-09-23T08:08:05.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456485 -le 1695456865 ]] 2023-09-23T08:08:05.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:08:05.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:08:05.337 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:15.346 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:15.346 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:15.346 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:08:15.347 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:16.437 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:08:16.437 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:08:16.438 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456496 2023-09-23T08:08:16.438 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456496 -le 1695456865 ]] 2023-09-23T08:08:16.438 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:08:16.439 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:08:16.439 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:20.543 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.533+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:08:20.543 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.533+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:08:20.545 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.540+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:08:20.545 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.543+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:08:20.545 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.543+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:08:20.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:20.545+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:08:26.447 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:26.447 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:26.447 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:26.463 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:08:27.476 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:08:27.477 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:08:27.477 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456507 2023-09-23T08:08:27.478 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456507 -le 1695456865 ]] 2023-09-23T08:08:27.478 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:08:27.478 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:08:27.478 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:36.691 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:08:36.691 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:08:36.691 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:08:36.691 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:08:36.691 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:08:36.692 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:36.685+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:08:37.480 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:37.487 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:37.487 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:37.520 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:08:38.481 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:08:38.482 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:08:38.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456518 2023-09-23T08:08:38.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456518 -le 1695456865 ]] 2023-09-23T08:08:38.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:08:38.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:08:38.484 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:46.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:08:46.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:08:46.638 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:08:46.638 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:08:46.638 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:08:46.639 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:08:46.639 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:08:46.639 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:08:46.635+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:08:48.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:48.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:48.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:08:48.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:49.649 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:08:49.650 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:08:49.651 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456529 2023-09-23T08:08:49.651 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456529 -le 1695456865 ]] 2023-09-23T08:08:49.651 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:08:49.651 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:08:49.651 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:08:59.658 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:08:59.661 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:08:59.664 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:08:59.672 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:00.787 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:00.790 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:00.793 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456540 2023-09-23T08:09:00.793 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456540 -le 1695456865 ]] 2023-09-23T08:09:00.793 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:00.793 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:00.794 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:09:10.801 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:09:10.801 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:10.801 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:09:10.805 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:09:11.930 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:11.930 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:11.931 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456551 2023-09-23T08:09:11.931 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456551 -le 1695456865 ]] 2023-09-23T08:09:11.931 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:11.931 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:11.931 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:09:21.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:09:21.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:21.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:09:21.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:09:23.028 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:23.028 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:23.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456563 2023-09-23T08:09:23.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456563 -le 1695456865 ]] 2023-09-23T08:09:23.029 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:23.030 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:23.030 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:09:33.033 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:09:33.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:09:33.034 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:33.046 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:09:34.103 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456574 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456574 -le 1695456865 ]] 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:34.107 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:09:42.551 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:09:42.549+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:09:44.135 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:09:44.135 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:09:44.144 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:09:44.144 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:45.271 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:45.271 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:45.272 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456585 2023-09-23T08:09:45.272 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456585 -le 1695456865 ]] 2023-09-23T08:09:45.272 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:45.273 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:45.273 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:09:55.276 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:09:55.277 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:09:55.279 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:09:55.314 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:09:56.492 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:09:56.492 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:09:56.493 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456596 2023-09-23T08:09:56.493 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456596 -le 1695456865 ]] 2023-09-23T08:09:56.493 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:09:56.493 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:09:56.494 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:10:06.499 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:10:06.500 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:10:06.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:10:06.506 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:10:07.636 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:10:07.637 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:10:07.639 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456607 2023-09-23T08:10:07.639 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456607 -le 1695456865 ]] 2023-09-23T08:10:07.639 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:10:07.639 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:10:07.639 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:10:17.650 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:10:17.661 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:10:17.661 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:10:17.668 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:10:18.782 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:10:18.783 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:10:18.783 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456618 2023-09-23T08:10:18.783 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456618 -le 1695456865 ]] 2023-09-23T08:10:18.783 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:10:18.783 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:10:18.784 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:10:27.374 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:10:27.372+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:10:27.374 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:10:27.372+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:10:28.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:10:28.786 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:10:28.795 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:10:28.795 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:10:29.850 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:10:29.850 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:10:29.851 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456629 2023-09-23T08:10:29.851 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456629 -le 1695456865 ]] 2023-09-23T08:10:29.851 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:10:29.851 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:10:29.851 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:10:39.866 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:10:39.867 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:10:39.867 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:10:39.897 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:10:40.943 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:10:40.944 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:10:40.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456640 2023-09-23T08:10:40.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456640 -le 1695456865 ]] 2023-09-23T08:10:40.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:10:40.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:10:40.945 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:10:50.955 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:10:50.956 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:10:50.959 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:10:50.959 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:10:51.981 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:10:51.984 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:10:51.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456651 2023-09-23T08:10:51.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456651 -le 1695456865 ]] 2023-09-23T08:10:51.985 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:10:51.986 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:10:51.986 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:01.995 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:01.995 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:01.996 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:01.996 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:03.168 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:03.169 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:03.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456663 2023-09-23T08:11:03.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456663 -le 1695456865 ]] 2023-09-23T08:11:03.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:03.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:03.170 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:13.176 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:13.177 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:13.177 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:13.177 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:14.305 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456674 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456674 -le 1695456865 ]] 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:14.306 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:24.313 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:24.313 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:24.319 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:24.319 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:25.406 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:25.407 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:25.409 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456685 2023-09-23T08:11:25.409 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456685 -le 1695456865 ]] 2023-09-23T08:11:25.409 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:25.409 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:25.409 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:35.415 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:35.415 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:35.422 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:35.422 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:36.551 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456696 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456696 -le 1695456865 ]] 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:36.552 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:46.559 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:46.560 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:46.574 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:46.575 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:47.716 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:47.716 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:47.725 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456707 2023-09-23T08:11:47.725 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456707 -le 1695456865 ]] 2023-09-23T08:11:47.726 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:47.726 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:47.726 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:11:57.743 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:11:57.743 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:11:57.766 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:11:57.769 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:11:58.838 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:11:58.840 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:11:58.840 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456718 2023-09-23T08:11:58.840 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456718 -le 1695456865 ]] 2023-09-23T08:11:58.841 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:11:58.841 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:11:58.841 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:12:08.848 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:12:08.849 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:12:08.849 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:12:08.850 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:12:09.968 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:12:09.969 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:12:09.970 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456729 2023-09-23T08:12:09.970 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456729 -le 1695456865 ]] 2023-09-23T08:12:09.970 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:12:09.970 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:12:09.970 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:12:19.978 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:12:19.978 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:12:19.978 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:12:19.979 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:12:20.981 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:12:20.981 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:12:20.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456740 2023-09-23T08:12:20.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456740 -le 1695456865 ]] 2023-09-23T08:12:20.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:12:20.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:12:20.982 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:12:31.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:12:31.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:12:31.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:12:31.001 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:12:32.047 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:12:32.047 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:12:32.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456752 2023-09-23T08:12:32.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456752 -le 1695456865 ]] 2023-09-23T08:12:32.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:12:32.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:12:32.048 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:12:34.472 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:12:34.469+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:12:34.472 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:12:34.469+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:12:42.063 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:12:42.063 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:12:42.064 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:12:42.074 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:12:43.067 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456763 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456763 -le 1695456865 ]] 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:12:43.074 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:12:53.077 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:12:53.077 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:12:53.077 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:12:53.080 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:12:54.219 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:12:54.226 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:12:54.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456774 2023-09-23T08:12:54.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456774 -le 1695456865 ]] 2023-09-23T08:12:54.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:12:54.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:12:54.227 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:04.234 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:04.234 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:13:04.234 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:04.237 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:05.373 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:13:05.375 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:13:05.376 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456785 2023-09-23T08:13:05.376 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456785 -le 1695456865 ]] 2023-09-23T08:13:05.376 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:13:05.376 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:13:05.376 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:15.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:15.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:13:15.378 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:15.379 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:16.343 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:13:16.343 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:13:16.344 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456796 2023-09-23T08:13:16.344 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456796 -le 1695456865 ]] 2023-09-23T08:13:16.344 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:13:16.345 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:13:16.345 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:18.041 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.039+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:13:18.041 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.039+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:13:18.042 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.039+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:13:18.042 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.039+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:13:18.042 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.039+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:13:18.047 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:13:18.047 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:13:18.047 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:13:18.048 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:18.045+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:13:26.348 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:26.354 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:26.354 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:26.355 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:13:27.504 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:13:27.505 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:13:27.506 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456807 2023-09-23T08:13:27.506 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456807 -le 1695456865 ]] 2023-09-23T08:13:27.506 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:13:27.506 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:13:27.506 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:37.507 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:13:37.508 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:37.509 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:37.514 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:38.647 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:13:38.649 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:13:38.649 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456818 2023-09-23T08:13:38.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456818 -le 1695456865 ]] 2023-09-23T08:13:38.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:13:38.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:13:38.650 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:40.822 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:40.818+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:13:40.832 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:40.821+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:13:40.832 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:13:40.821+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:13:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:13:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:48.659 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:49.794 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:13:49.794 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:13:49.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456829 2023-09-23T08:13:49.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456829 -le 1695456865 ]] 2023-09-23T08:13:49.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:13:49.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:13:49.795 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:13:59.798 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:13:59.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:13:59.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:13:59.799 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:14:00.913 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:14:00.914 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:14:00.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456840 2023-09-23T08:14:00.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456840 -le 1695456865 ]] 2023-09-23T08:14:00.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:14:00.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:14:00.916 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:14:10.924 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:14:10.924 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:14:10.924 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:14:10.933 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:14:12.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:14:12.144 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:14:12.145 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456852 2023-09-23T08:14:12.145 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456852 -le 1695456865 ]] 2023-09-23T08:14:12.145 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:14:12.145 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:14:12.145 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:14:18.313 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:14:18.309+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:14:18.314 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:14:18.309+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:14:22.151 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:14:22.151 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:14:22.160 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:14:22.168 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:14:23.285 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:14:23.285 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:14:23.286 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456863 2023-09-23T08:14:23.286 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456863 -le 1695456865 ]] 2023-09-23T08:14:23.286 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ ! -z 172.21.15.140:0/1974081988 ]] 2023-09-23T08:14:23.286 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 172.21.15.140:0/1974081988 != \1\7\2\.\2\1\.\1\5\.\1\4\0\:\0\/\1\9\7\4\0\8\1\9\8\8 ]] 2023-09-23T08:14:23.286 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:14:33.288 INFO:tasks.workunit.client.0.smithi140.stderr:++ ceph mgr dump 2023-09-23T08:14:33.294 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T08:14:33.294 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T08:14:33.294 INFO:tasks.workunit.client.0.smithi140.stderr:++ jq '.active_clients[]' 2023-09-23T08:14:34.412 INFO:tasks.workunit.client.0.smithi140.stderr:+ CLIENT_ADDR=172.21.15.140:0/1974081988 2023-09-23T08:14:34.413 INFO:tasks.workunit.client.0.smithi140.stderr:++ date +%s 2023-09-23T08:14:34.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ CURRENT_TIME=1695456874 2023-09-23T08:14:34.414 INFO:tasks.workunit.client.0.smithi140.stderr:+ [[ 1695456874 -le 1695456865 ]] 2023-09-23T08:14:34.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i = 1 )) 2023-09-23T08:14:34.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:14:34.479 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:14:35.095 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:14:35.093+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:14:35.095 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:14:35.097 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:14:45.099 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:14:45.099 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:14:45.099 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:14:45.261 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:14:45.265 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:14:45.265 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:14:45.260+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:14:55.267 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:14:55.268 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:14:55.268 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:14:55.373 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:14:55.370+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:14:55.373 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:14:55.402 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:05.404 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:05.404 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:05.404 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:05.513 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:05.511+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:05.514 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:05.542 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:15.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:15.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:15.553 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:15.731 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:15.728+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:15.736 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:15.751 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:25.752 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:25.752 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:25.752 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:25.826 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:25.824+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:25.826 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:25.845 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:35.852 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:35.852 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:35.853 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:35.939 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:35.937+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:35.940 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:35.953 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:45.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:45.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:45.956 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:46.074 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:46.072+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:47.069 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:47.067+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:47.070 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:47.079 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:15:57.087 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:15:57.087 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:15:57.087 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:15:57.344 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:15:57.345 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:15:57.339+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:15:57.363 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:07.365 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:07.365 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:07.365 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:07.450 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:07.447+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:07.453 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:07.489 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:11.985 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:16:11.979+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:16:11.985 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:16:11.979+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:16:17.494 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:17.494 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:17.494 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:17.607 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:17.605+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:17.610 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:17.640 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:27.641 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:27.641 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:27.642 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:27.770 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:27.768+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:27.771 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:27.774 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:30.182 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:16:30.180+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:16:30.183 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:16:30.180+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:16:37.779 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:37.780 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:37.780 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:37.897 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:37.894+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:37.897 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:37.920 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:47.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:47.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:47.924 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:48.039 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:48.036+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:48.039 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:48.057 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:16:58.058 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:16:58.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:16:58.059 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:16:58.150 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:16:58.148+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:16:58.151 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:16:58.171 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:08.173 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:08.173 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:08.173 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:08.238 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:08.235+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:08.238 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:08.253 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:18.258 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:18.259 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:18.259 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:18.375 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:18.373+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:18.376 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:18.383 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:28.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:28.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:28.385 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:28.492 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:28.489+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:28.492 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:28.529 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:38.545 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:38.545 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:38.545 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:38.633 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:38.634 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:38.632+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:38.655 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:48.657 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:48.735 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:48.733+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:48.740 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:48.765 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:17:58.767 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:17:58.767 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:17:58.767 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:17:58.873 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:17:58.870+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:17:58.874 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:17:58.894 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:18:08.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:18:08.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:18:08.898 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:18:08.992 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:18:08.989+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:18:08.992 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:18:09.020 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:18:19.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:18:19.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:18:19.022 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:18:19.119 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:18:19.116+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:18:19.123 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:18:19.142 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:18:29.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:18:29.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:18:29.144 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T08:18:29.219 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:18:29.217+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:18:29.225 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:18:29.242 INFO:tasks.workunit.client.0.smithi140.stderr:+ sleep 10 2023-09-23T08:18:39.246 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i++ )) 2023-09-23T08:18:39.246 INFO:tasks.workunit.client.0.smithi140.stderr:+ (( i <= 24 )) 2023-09-23T08:18:39.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-09-23T08:18:39.319 INFO:tasks.workunit.client.0.smithi140.stderr:+ grep 'every 2m' 2023-09-23T08:18:39.402 INFO:tasks.ceph.mgr.x.smithi140.stderr:2023-09-23T08:18:39.401+0000 7f556455d700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T08:18:39.405 INFO:tasks.workunit.client.0.smithi140.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T08:18:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T08:18:39.435 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-09-23T08:18:39.435 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-23T08:18:41.596 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-23T08:18:41.664 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5731a07b317e40dc8607ec4c9d013822 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-23T08:18:41.667 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-23T08:18:41.725 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-23T08:18:41.746 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-23T08:18:41.746 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:18:42.375 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:18:42.382 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:18:42.402 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2091,"stamp":"2023-09-23T08:18:40.920041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19943962204,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1077857,"num_read_kb":3973992,"num_write":1003690,"num_write_kb":3956925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75261,"ondisk_log_size":75261,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17253216,"kb_used_data":11869176,"kb_used_omap":1733,"kb_used_meta":5382266,"kb_avail":265862304,"statfs":{"total":289910292480,"available":272242999296,"internally_reserved":0,"allocated":12154036224,"data_stored":54478303515,"data_compressed":36588590,"data_compressed_allocated":1683861504,"data_compressed_original":42588610560,"omap_allocated":1775190,"internal_metadata":5511440810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,5,10,17,13,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8007680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2172,"num_read_kb":8620,"num_write":2262,"num_write_kb":9048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.012255"},"pg_stats":[{"pgid":"2.7","version":"92'5283","reported_seq":8862,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689496+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T08:18:35.689496+0000","last_peered":"2023-09-23T08:18:35.689496+0000","last_clean":"2023-09-23T08:18:35.689496+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:18:35.689496+0000","last_undegraded":"2023-09-23T08:18:35.689496+0000","last_fullsized":"2023-09-23T08:18:35.689496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:22:47.585270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'7751","reported_seq":13389,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690123+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T08:18:35.690123+0000","last_peered":"2023-09-23T08:18:35.690123+0000","last_clean":"2023-09-23T08:18:35.690123+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:18:35.690123+0000","last_undegraded":"2023-09-23T08:18:35.690123+0000","last_fullsized":"2023-09-23T08:18:35.690123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:46.793786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3765","reported_seq":6160,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690922+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T08:18:35.690922+0000","last_peered":"2023-09-23T08:18:35.690922+0000","last_clean":"2023-09-23T08:18:35.690922+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:18:35.690922+0000","last_undegraded":"2023-09-23T08:18:35.690922+0000","last_fullsized":"2023-09-23T08:18:35.690922+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:45:34.095014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'252711","reported_seq":506164,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:36.855853+0000","last_change":"2023-09-23T07:35:03.039194+0000","last_active":"2023-09-23T08:18:36.855853+0000","last_peered":"2023-09-23T08:18:36.855853+0000","last_clean":"2023-09-23T08:18:36.855853+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:18:36.855853+0000","last_undegraded":"2023-09-23T08:18:36.855853+0000","last_fullsized":"2023-09-23T08:18:36.855853+0000","mapping_epoch":18,"log_start":"90'242663","ondisk_log_start":"90'242663","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:10:33.493561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5097197568,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":254657,"num_read_kb":997024,"num_write":251329,"num_write_kb":1005316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"88'509","reported_seq":2897,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.701281+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T08:18:35.701281+0000","last_peered":"2023-09-23T08:18:35.701281+0000","last_clean":"2023-09-23T08:18:35.701281+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:18:35.701281+0000","last_undegraded":"2023-09-23T08:18:35.701281+0000","last_fullsized":"2023-09-23T08:18:35.701281+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:26:43.861575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'239046","reported_seq":479283,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.891100+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:18:37.891100+0000","last_peered":"2023-09-23T08:18:37.891100+0000","last_clean":"2023-09-23T08:18:37.891100+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:18:37.891100+0000","last_undegraded":"2023-09-23T08:18:37.891100+0000","last_fullsized":"2023-09-23T08:18:37.891100+0000","mapping_epoch":18,"log_start":"90'229022","ondisk_log_start":"90'229022","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:22:37.365482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4830466067,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":241360,"num_read_kb":945388,"num_write":237745,"num_write_kb":950974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":250,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-09-23T08:17:25.381867+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T08:17:25.381867+0000","last_peered":"2023-09-23T08:17:25.381867+0000","last_clean":"2023-09-23T08:17:25.381867+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:17:25.381867+0000","last_undegraded":"2023-09-23T08:17:25.381867+0000","last_fullsized":"2023-09-23T08:17:25.381867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:01:08.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'6443","reported_seq":9964,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689292+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T08:18:35.689292+0000","last_peered":"2023-09-23T08:18:35.689292+0000","last_clean":"2023-09-23T08:18:35.689292+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:18:35.689292+0000","last_undegraded":"2023-09-23T08:18:35.689292+0000","last_fullsized":"2023-09-23T08:18:35.689292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:02:47.322360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'240689","reported_seq":482548,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.882872+0000","last_change":"2023-09-23T07:35:03.043707+0000","last_active":"2023-09-23T08:18:37.882872+0000","last_peered":"2023-09-23T08:18:37.882872+0000","last_clean":"2023-09-23T08:18:37.882872+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:18:37.882872+0000","last_undegraded":"2023-09-23T08:18:37.882872+0000","last_fullsized":"2023-09-23T08:18:37.882872+0000","mapping_epoch":18,"log_start":"90'230622","ondisk_log_start":"90'230622","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:17:16.184796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4848480256,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":242973,"num_read_kb":951672,"num_write":239397,"num_write_kb":957588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'2310","reported_seq":3578,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.453008+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T08:18:35.453008+0000","last_peered":"2023-09-23T08:18:35.453008+0000","last_clean":"2023-09-23T08:18:35.453008+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:18:35.453008+0000","last_undegraded":"2023-09-23T08:18:35.453008+0000","last_fullsized":"2023-09-23T08:18:35.453008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:23:50.018321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'256878","reported_seq":514028,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:36.914543+0000","last_change":"2023-09-23T07:35:03.045902+0000","last_active":"2023-09-23T08:18:36.914543+0000","last_peered":"2023-09-23T08:18:36.914543+0000","last_clean":"2023-09-23T08:18:36.914543+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:18:36.914543+0000","last_undegraded":"2023-09-23T08:18:36.914543+0000","last_fullsized":"2023-09-23T08:18:36.914543+0000","mapping_epoch":18,"log_start":"90'246864","ondisk_log_start":"90'246864","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:55:26.251473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5167218688,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":258331,"num_read_kb":1012144,"num_write":255521,"num_write_kb":1022084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'4501","reported_seq":6876,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.691641+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T08:18:35.691641+0000","last_peered":"2023-09-23T08:18:35.691641+0000","last_clean":"2023-09-23T08:18:35.691641+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:18:35.691641+0000","last_undegraded":"2023-09-23T08:18:35.691641+0000","last_fullsized":"2023-09-23T08:18:35.691641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:29:11.408969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"92'4449","reported_seq":8555,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690598+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T08:18:35.690598+0000","last_peered":"2023-09-23T08:18:35.690598+0000","last_clean":"2023-09-23T08:18:35.690598+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:18:35.690598+0000","last_undegraded":"2023-09-23T08:18:35.690598+0000","last_fullsized":"2023-09-23T08:18:35.690598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:45.472429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19943362579,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":997321,"num_read_kb":3906228,"num_write":983992,"num_write_kb":3935962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12151652352,"data_stored":54476107776,"data_compressed":36583527,"data_compressed_allocated":1683795968,"data_compressed_original":42586238976,"omap_allocated":0,"internal_metadata":0},"log_size":40153,"ondisk_log_size":40153,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35011,"ondisk_log_size":35011,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5665368,"kb_used_data":3956432,"kb_used_omap":572,"kb_used_meta":1708355,"kb_avail":88706472,"statfs":{"total":96636764160,"available":90835427328,"internally_reserved":0,"allocated":4051386368,"data_stored":18159039233,"data_compressed":11795809,"data_compressed_allocated":561262592,"data_compressed_original":14195621888,"omap_allocated":585792,"internal_metadata":1749356480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":2.0110000000000001,"15min":2.1899999999999999},"min":{"1min":0.27300000000000002,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.0830000000000002,"5min":13.019,"15min":16.388999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.284,"5min":2.2799999999999998,"15min":2.5390000000000001},"min":{"1min":0.29899999999999999,"5min":0.219,"15min":0.185},"max":{"1min":5.1109999999999998,"5min":22.266999999999999,"15min":22.266999999999999},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":3.165,"5min":3.456,"15min":3.2200000000000002},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.186},"max":{"1min":11.971,"5min":19.433,"15min":23.045999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.7130000000000001,"5min":2.827,"15min":3.048},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.92,"5min":11.92,"15min":23.100999999999999},"last":1.2270000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739165,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5755328,"kb_used_data":3955944,"kb_used_omap":579,"kb_used_meta":1798780,"kb_avail":88616512,"statfs":{"total":96636764160,"available":90743308288,"internally_reserved":0,"allocated":4050886656,"data_stored":18159633998,"data_compressed":12385007,"data_compressed_allocated":561299456,"data_compressed_original":14196805632,"omap_allocated":593087,"internal_metadata":1841951553},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,12,7,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":2.4980000000000002,"15min":2.7229999999999999},"min":{"1min":0.24099999999999999,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":6.5410000000000004,"5min":11.981999999999999,"15min":19.885999999999999},"last":2.5190000000000001},{"interface":"front","average":{"1min":2.867,"5min":3.4430000000000001,"15min":2.9830000000000001},"min":{"1min":0.22900000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.888999999999999,"5min":14.66,"15min":24.385000000000002},"last":1.8180000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":3.194,"5min":3.7719999999999998,"15min":3.274},"min":{"1min":0.33000000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.9760000000000009,"5min":25.84,"15min":25.84},"last":3.0859999999999999},{"interface":"front","average":{"1min":2.1509999999999998,"5min":2.77,"15min":2.7949999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.5499999999999998,"5min":25.120999999999999,"15min":25.120999999999999},"last":0.82899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5832520,"kb_used_data":3956800,"kb_used_omap":582,"kb_used_meta":1875129,"kb_avail":88539320,"statfs":{"total":96636764160,"available":90664263680,"internally_reserved":0,"allocated":4051763200,"data_stored":18159630284,"data_compressed":12407774,"data_compressed_allocated":561299456,"data_compressed_original":14196183040,"omap_allocated":596311,"internal_metadata":1920132777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,5,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.5110000000000001,"5min":2.5939999999999999,"15min":2.5699999999999998},"min":{"1min":0.24399999999999999,"5min":0.161,"15min":0.14699999999999999},"max":{"1min":7.2779999999999996,"5min":13.217000000000001,"15min":14.496},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.1240000000000001,"5min":3.4039999999999999,"15min":3.4430000000000001},"min":{"1min":0.28299999999999997,"5min":0.192,"15min":0.188},"max":{"1min":12.435,"5min":14.779,"15min":25.012},"last":0.20300000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.6909999999999998,"5min":3.5059999999999998,"15min":3.1549999999999998},"min":{"1min":0.28199999999999997,"5min":0.19,"15min":0.19},"max":{"1min":11.538,"5min":20.216999999999999,"15min":23.443999999999999},"last":18.414000000000001},{"interface":"front","average":{"1min":1.677,"5min":1.5940000000000001,"15min":2.0430000000000001},"min":{"1min":0.255,"5min":0.193,"15min":0.187},"max":{"1min":8.7560000000000002,"5min":10.983000000000001,"15min":20.565000000000001},"last":18.353000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4050063360,"data_stored":18158690304,"data_compressed":12382449,"data_compressed_allocated":561262592,"data_compressed_original":14195855360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4050792448,"data_stored":18158706688,"data_compressed":12405009,"data_compressed_allocated":561266688,"data_compressed_original":14195191808,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4050796544,"data_stored":18158710784,"data_compressed":11796069,"data_compressed_allocated":561266688,"data_compressed_original":14195191808,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:18:42.404 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:18:43.044 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:18:43.080 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:18:43.104 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2092,"stamp":"2023-09-23T08:18:42.920393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19946002012,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1078438,"num_read_kb":3976308,"num_write":1004246,"num_write_kb":3959149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75308,"ondisk_log_size":75308,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17286284,"kb_used_data":11875892,"kb_used_omap":1733,"kb_used_meta":5408634,"kb_avail":265829236,"statfs":{"total":289910292480,"available":272209137664,"internally_reserved":0,"allocated":12160913408,"data_stored":54478623003,"data_compressed":36590312,"data_compressed_allocated":1683943424,"data_compressed_original":42582503424,"omap_allocated":1775190,"internal_metadata":5538441642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,1,1,15,17,16,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6074368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1714,"num_read_kb":6820,"num_write":1688,"num_write_kb":6752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.012245"},"pg_stats":[{"pgid":"2.7","version":"92'5283","reported_seq":8862,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689496+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T08:18:35.689496+0000","last_peered":"2023-09-23T08:18:35.689496+0000","last_clean":"2023-09-23T08:18:35.689496+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:18:35.689496+0000","last_undegraded":"2023-09-23T08:18:35.689496+0000","last_fullsized":"2023-09-23T08:18:35.689496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:22:47.585270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'7751","reported_seq":13389,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690123+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T08:18:35.690123+0000","last_peered":"2023-09-23T08:18:35.690123+0000","last_clean":"2023-09-23T08:18:35.690123+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:18:35.690123+0000","last_undegraded":"2023-09-23T08:18:35.690123+0000","last_fullsized":"2023-09-23T08:18:35.690123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:46.793786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3765","reported_seq":6160,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690922+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T08:18:35.690922+0000","last_peered":"2023-09-23T08:18:35.690922+0000","last_clean":"2023-09-23T08:18:35.690922+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:18:35.690922+0000","last_undegraded":"2023-09-23T08:18:35.690922+0000","last_fullsized":"2023-09-23T08:18:35.690922+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:45:34.095014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'253003","reported_seq":506737,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:41.977332+0000","last_change":"2023-09-23T07:35:03.039194+0000","last_active":"2023-09-23T08:18:41.977332+0000","last_peered":"2023-09-23T08:18:41.977332+0000","last_clean":"2023-09-23T08:18:41.977332+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:18:41.977332+0000","last_undegraded":"2023-09-23T08:18:41.977332+0000","last_fullsized":"2023-09-23T08:18:41.977332+0000","mapping_epoch":18,"log_start":"90'242971","ondisk_log_start":"90'242971","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:10:33.493561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5098274816,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":254938,"num_read_kb":998144,"num_write":251621,"num_write_kb":1006484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"88'509","reported_seq":2899,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:40.815814+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T08:18:40.815814+0000","last_peered":"2023-09-23T08:18:40.815814+0000","last_clean":"2023-09-23T08:18:40.815814+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:18:40.815814+0000","last_undegraded":"2023-09-23T08:18:40.815814+0000","last_fullsized":"2023-09-23T08:18:40.815814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:26:43.861575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'239046","reported_seq":479283,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.891100+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:18:37.891100+0000","last_peered":"2023-09-23T08:18:37.891100+0000","last_clean":"2023-09-23T08:18:37.891100+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:18:37.891100+0000","last_undegraded":"2023-09-23T08:18:37.891100+0000","last_fullsized":"2023-09-23T08:18:37.891100+0000","mapping_epoch":18,"log_start":"90'229022","ondisk_log_start":"90'229022","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:22:37.365482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4830466067,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":241360,"num_read_kb":945388,"num_write":237745,"num_write_kb":950974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":250,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-09-23T08:17:25.381867+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T08:17:25.381867+0000","last_peered":"2023-09-23T08:17:25.381867+0000","last_clean":"2023-09-23T08:17:25.381867+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:17:25.381867+0000","last_undegraded":"2023-09-23T08:17:25.381867+0000","last_fullsized":"2023-09-23T08:17:25.381867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:01:08.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'6443","reported_seq":9964,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689292+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T08:18:35.689292+0000","last_peered":"2023-09-23T08:18:35.689292+0000","last_clean":"2023-09-23T08:18:35.689292+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:18:35.689292+0000","last_undegraded":"2023-09-23T08:18:35.689292+0000","last_fullsized":"2023-09-23T08:18:35.689292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:02:47.322360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'240689","reported_seq":482548,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.882872+0000","last_change":"2023-09-23T07:35:03.043707+0000","last_active":"2023-09-23T08:18:37.882872+0000","last_peered":"2023-09-23T08:18:37.882872+0000","last_clean":"2023-09-23T08:18:37.882872+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:18:37.882872+0000","last_undegraded":"2023-09-23T08:18:37.882872+0000","last_fullsized":"2023-09-23T08:18:37.882872+0000","mapping_epoch":18,"log_start":"90'230622","ondisk_log_start":"90'230622","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:17:16.184796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4848480256,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":242973,"num_read_kb":951672,"num_write":239397,"num_write_kb":957588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'2310","reported_seq":3579,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.695820+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T08:18:35.695820+0000","last_peered":"2023-09-23T08:18:35.695820+0000","last_clean":"2023-09-23T08:18:35.695820+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:18:35.695820+0000","last_undegraded":"2023-09-23T08:18:35.695820+0000","last_fullsized":"2023-09-23T08:18:35.695820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:23:50.018321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'257142","reported_seq":514591,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:41.955802+0000","last_change":"2023-09-23T07:35:03.045902+0000","last_active":"2023-09-23T08:18:41.955802+0000","last_peered":"2023-09-23T08:18:41.955802+0000","last_clean":"2023-09-23T08:18:41.955802+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:18:41.955802+0000","last_undegraded":"2023-09-23T08:18:41.955802+0000","last_fullsized":"2023-09-23T08:18:41.955802+0000","mapping_epoch":18,"log_start":"90'247069","ondisk_log_start":"90'247069","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:55:26.251473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5168181248,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":258631,"num_read_kb":1013340,"num_write":255785,"num_write_kb":1023140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4505","reported_seq":6882,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:40.848742+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T08:18:40.848742+0000","last_peered":"2023-09-23T08:18:40.848742+0000","last_clean":"2023-09-23T08:18:40.848742+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:18:40.848742+0000","last_undegraded":"2023-09-23T08:18:40.848742+0000","last_fullsized":"2023-09-23T08:18:40.848742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:29:11.408969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"92'4449","reported_seq":8555,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690598+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T08:18:35.690598+0000","last_peered":"2023-09-23T08:18:35.690598+0000","last_clean":"2023-09-23T08:18:35.690598+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:18:35.690598+0000","last_undegraded":"2023-09-23T08:18:35.690598+0000","last_fullsized":"2023-09-23T08:18:35.690598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:45.472429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19945402387,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":997902,"num_read_kb":3908544,"num_write":984548,"num_write_kb":3938186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12158615552,"data_stored":54476308480,"data_compressed":36584122,"data_compressed_allocated":1683881984,"data_compressed_original":42579943424,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35015,"ondisk_log_size":35015,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5690920,"kb_used_data":3959776,"kb_used_omap":572,"kb_used_meta":1730563,"kb_avail":88680920,"statfs":{"total":96636764160,"available":90809262080,"internally_reserved":0,"allocated":4054810624,"data_stored":18159211265,"data_compressed":11796873,"data_compressed_allocated":561303552,"data_compressed_original":14192603136,"omap_allocated":585792,"internal_metadata":1772097472},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":2.0110000000000001,"15min":2.1899999999999999},"min":{"1min":0.27300000000000002,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.0830000000000002,"5min":13.019,"15min":16.388999999999999},"last":11.561},{"interface":"front","average":{"1min":1.284,"5min":2.2799999999999998,"15min":2.5390000000000001},"min":{"1min":0.29899999999999999,"5min":0.219,"15min":0.185},"max":{"1min":5.1109999999999998,"5min":22.266999999999999,"15min":22.266999999999999},"last":11.597}]},{"osd":1,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":3.165,"5min":3.456,"15min":3.2200000000000002},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.186},"max":{"1min":11.971,"5min":19.433,"15min":23.045999999999999},"last":3.5110000000000001},{"interface":"front","average":{"1min":2.7130000000000001,"5min":2.827,"15min":3.048},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.92,"5min":11.92,"15min":23.100999999999999},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739166,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5762844,"kb_used_data":3959316,"kb_used_omap":579,"kb_used_meta":1802940,"kb_avail":88608996,"statfs":{"total":96636764160,"available":90735611904,"internally_reserved":0,"allocated":4054339584,"data_stored":18159781454,"data_compressed":12385665,"data_compressed_allocated":561340416,"data_compressed_original":14193717248,"omap_allocated":593087,"internal_metadata":1846211393},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,10,8,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":2.4980000000000002,"15min":2.7229999999999999},"min":{"1min":0.24099999999999999,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":6.5410000000000004,"5min":11.981999999999999,"15min":19.885999999999999},"last":4.2300000000000004},{"interface":"front","average":{"1min":2.867,"5min":3.4430000000000001,"15min":2.9830000000000001},"min":{"1min":0.22900000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.888999999999999,"5min":14.66,"15min":24.385000000000002},"last":4.2800000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":3.194,"5min":3.7719999999999998,"15min":3.274},"min":{"1min":0.33000000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.9760000000000009,"5min":25.84,"15min":25.84},"last":4.5030000000000001},{"interface":"front","average":{"1min":2.1509999999999998,"5min":2.77,"15min":2.7949999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.5499999999999998,"5min":25.120999999999999,"15min":25.120999999999999},"last":4.2640000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5832520,"kb_used_data":3956800,"kb_used_omap":582,"kb_used_meta":1875129,"kb_avail":88539320,"statfs":{"total":96636764160,"available":90664263680,"internally_reserved":0,"allocated":4051763200,"data_stored":18159630284,"data_compressed":12407774,"data_compressed_allocated":561299456,"data_compressed_original":14196183040,"omap_allocated":596311,"internal_metadata":1920132777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,5,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.5110000000000001,"5min":2.5939999999999999,"15min":2.5699999999999998},"min":{"1min":0.24399999999999999,"5min":0.161,"15min":0.14699999999999999},"max":{"1min":7.2779999999999996,"5min":13.217000000000001,"15min":14.496},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.1240000000000001,"5min":3.4039999999999999,"15min":3.4430000000000001},"min":{"1min":0.28299999999999997,"5min":0.192,"15min":0.188},"max":{"1min":12.435,"5min":14.779,"15min":25.012},"last":0.20300000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.6909999999999998,"5min":3.5059999999999998,"15min":3.1549999999999998},"min":{"1min":0.28199999999999997,"5min":0.19,"15min":0.19},"max":{"1min":11.538,"5min":20.216999999999999,"15min":23.443999999999999},"last":18.414000000000001},{"interface":"front","average":{"1min":1.677,"5min":1.5940000000000001,"15min":2.0430000000000001},"min":{"1min":0.255,"5min":0.193,"15min":0.187},"max":{"1min":8.7560000000000002,"5min":10.983000000000001,"15min":20.565000000000001},"last":18.353000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4053647360,"data_stored":18158784512,"data_compressed":12382219,"data_compressed_allocated":561303552,"data_compressed_original":14192607232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4050792448,"data_stored":18158706688,"data_compressed":12405009,"data_compressed_allocated":561266688,"data_compressed_original":14195191808,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4054175744,"data_stored":18158817280,"data_compressed":11796894,"data_compressed_allocated":561311744,"data_compressed_original":14192144384,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:18:43.106 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-23T08:18:43.106 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:18:43.719 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:18:43.735 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:18:43.757 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2092,"stamp":"2023-09-23T08:18:42.920393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19946002012,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1078438,"num_read_kb":3976308,"num_write":1004246,"num_write_kb":3959149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75308,"ondisk_log_size":75308,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17286284,"kb_used_data":11875892,"kb_used_omap":1733,"kb_used_meta":5408634,"kb_avail":265829236,"statfs":{"total":289910292480,"available":272209137664,"internally_reserved":0,"allocated":12160913408,"data_stored":54478623003,"data_compressed":36590312,"data_compressed_allocated":1683943424,"data_compressed_original":42582503424,"omap_allocated":1775190,"internal_metadata":5538441642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,1,1,15,17,16,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6074368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1714,"num_read_kb":6820,"num_write":1688,"num_write_kb":6752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.012245"},"pg_stats":[{"pgid":"2.7","version":"92'5283","reported_seq":8862,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689496+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T08:18:35.689496+0000","last_peered":"2023-09-23T08:18:35.689496+0000","last_clean":"2023-09-23T08:18:35.689496+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:18:35.689496+0000","last_undegraded":"2023-09-23T08:18:35.689496+0000","last_fullsized":"2023-09-23T08:18:35.689496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T16:22:47.585270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'7751","reported_seq":13389,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690123+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T08:18:35.690123+0000","last_peered":"2023-09-23T08:18:35.690123+0000","last_clean":"2023-09-23T08:18:35.690123+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:18:35.690123+0000","last_undegraded":"2023-09-23T08:18:35.690123+0000","last_fullsized":"2023-09-23T08:18:35.690123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:46.793786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3765","reported_seq":6160,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690922+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T08:18:35.690922+0000","last_peered":"2023-09-23T08:18:35.690922+0000","last_clean":"2023-09-23T08:18:35.690922+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:18:35.690922+0000","last_undegraded":"2023-09-23T08:18:35.690922+0000","last_fullsized":"2023-09-23T08:18:35.690922+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:45:34.095014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'253003","reported_seq":506737,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:41.977332+0000","last_change":"2023-09-23T07:35:03.039194+0000","last_active":"2023-09-23T08:18:41.977332+0000","last_peered":"2023-09-23T08:18:41.977332+0000","last_clean":"2023-09-23T08:18:41.977332+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:18:41.977332+0000","last_undegraded":"2023-09-23T08:18:41.977332+0000","last_fullsized":"2023-09-23T08:18:41.977332+0000","mapping_epoch":18,"log_start":"90'242971","ondisk_log_start":"90'242971","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:10:33.493561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5098274816,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":254938,"num_read_kb":998144,"num_write":251621,"num_write_kb":1006484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"88'509","reported_seq":2899,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:40.815814+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T08:18:40.815814+0000","last_peered":"2023-09-23T08:18:40.815814+0000","last_clean":"2023-09-23T08:18:40.815814+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:18:40.815814+0000","last_undegraded":"2023-09-23T08:18:40.815814+0000","last_fullsized":"2023-09-23T08:18:40.815814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:26:43.861575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'239046","reported_seq":479283,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.891100+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:18:37.891100+0000","last_peered":"2023-09-23T08:18:37.891100+0000","last_clean":"2023-09-23T08:18:37.891100+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:18:37.891100+0000","last_undegraded":"2023-09-23T08:18:37.891100+0000","last_fullsized":"2023-09-23T08:18:37.891100+0000","mapping_epoch":18,"log_start":"90'229022","ondisk_log_start":"90'229022","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:22:37.365482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4830466067,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":241360,"num_read_kb":945388,"num_write":237745,"num_write_kb":950974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":250,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-09-23T08:17:25.381867+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T08:17:25.381867+0000","last_peered":"2023-09-23T08:17:25.381867+0000","last_clean":"2023-09-23T08:17:25.381867+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:17:25.381867+0000","last_undegraded":"2023-09-23T08:17:25.381867+0000","last_fullsized":"2023-09-23T08:17:25.381867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:01:08.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'6443","reported_seq":9964,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.689292+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T08:18:35.689292+0000","last_peered":"2023-09-23T08:18:35.689292+0000","last_clean":"2023-09-23T08:18:35.689292+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:18:35.689292+0000","last_undegraded":"2023-09-23T08:18:35.689292+0000","last_fullsized":"2023-09-23T08:18:35.689292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:02:47.322360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'240689","reported_seq":482548,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:37.882872+0000","last_change":"2023-09-23T07:35:03.043707+0000","last_active":"2023-09-23T08:18:37.882872+0000","last_peered":"2023-09-23T08:18:37.882872+0000","last_clean":"2023-09-23T08:18:37.882872+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:18:37.882872+0000","last_undegraded":"2023-09-23T08:18:37.882872+0000","last_fullsized":"2023-09-23T08:18:37.882872+0000","mapping_epoch":18,"log_start":"90'230622","ondisk_log_start":"90'230622","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:17:16.184796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4848480256,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":242973,"num_read_kb":951672,"num_write":239397,"num_write_kb":957588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'2310","reported_seq":3579,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.695820+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T08:18:35.695820+0000","last_peered":"2023-09-23T08:18:35.695820+0000","last_clean":"2023-09-23T08:18:35.695820+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:18:35.695820+0000","last_undegraded":"2023-09-23T08:18:35.695820+0000","last_fullsized":"2023-09-23T08:18:35.695820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:23:50.018321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'257142","reported_seq":514591,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:41.955802+0000","last_change":"2023-09-23T07:35:03.045902+0000","last_active":"2023-09-23T08:18:41.955802+0000","last_peered":"2023-09-23T08:18:41.955802+0000","last_clean":"2023-09-23T08:18:41.955802+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:18:41.955802+0000","last_undegraded":"2023-09-23T08:18:41.955802+0000","last_fullsized":"2023-09-23T08:18:41.955802+0000","mapping_epoch":18,"log_start":"90'247069","ondisk_log_start":"90'247069","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:55:26.251473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5168181248,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":258631,"num_read_kb":1013340,"num_write":255785,"num_write_kb":1023140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4505","reported_seq":6882,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:40.848742+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T08:18:40.848742+0000","last_peered":"2023-09-23T08:18:40.848742+0000","last_clean":"2023-09-23T08:18:40.848742+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:18:40.848742+0000","last_undegraded":"2023-09-23T08:18:40.848742+0000","last_fullsized":"2023-09-23T08:18:40.848742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:29:11.408969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"92'4449","reported_seq":8555,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.690598+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T08:18:35.690598+0000","last_peered":"2023-09-23T08:18:35.690598+0000","last_clean":"2023-09-23T08:18:35.690598+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:18:35.690598+0000","last_undegraded":"2023-09-23T08:18:35.690598+0000","last_fullsized":"2023-09-23T08:18:35.690598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:45.472429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19945402387,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":997902,"num_read_kb":3908544,"num_write":984548,"num_write_kb":3938186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12158615552,"data_stored":54476308480,"data_compressed":36584122,"data_compressed_allocated":1683881984,"data_compressed_original":42579943424,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35015,"ondisk_log_size":35015,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5690920,"kb_used_data":3959776,"kb_used_omap":572,"kb_used_meta":1730563,"kb_avail":88680920,"statfs":{"total":96636764160,"available":90809262080,"internally_reserved":0,"allocated":4054810624,"data_stored":18159211265,"data_compressed":11796873,"data_compressed_allocated":561303552,"data_compressed_original":14192603136,"omap_allocated":585792,"internal_metadata":1772097472},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":2.0110000000000001,"15min":2.1899999999999999},"min":{"1min":0.27300000000000002,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.0830000000000002,"5min":13.019,"15min":16.388999999999999},"last":11.561},{"interface":"front","average":{"1min":1.284,"5min":2.2799999999999998,"15min":2.5390000000000001},"min":{"1min":0.29899999999999999,"5min":0.219,"15min":0.185},"max":{"1min":5.1109999999999998,"5min":22.266999999999999,"15min":22.266999999999999},"last":11.597}]},{"osd":1,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":3.165,"5min":3.456,"15min":3.2200000000000002},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.186},"max":{"1min":11.971,"5min":19.433,"15min":23.045999999999999},"last":3.5110000000000001},{"interface":"front","average":{"1min":2.7130000000000001,"5min":2.827,"15min":3.048},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.92,"5min":11.92,"15min":23.100999999999999},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739166,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5762844,"kb_used_data":3959316,"kb_used_omap":579,"kb_used_meta":1802940,"kb_avail":88608996,"statfs":{"total":96636764160,"available":90735611904,"internally_reserved":0,"allocated":4054339584,"data_stored":18159781454,"data_compressed":12385665,"data_compressed_allocated":561340416,"data_compressed_original":14193717248,"omap_allocated":593087,"internal_metadata":1846211393},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,10,8,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":2.4980000000000002,"15min":2.7229999999999999},"min":{"1min":0.24099999999999999,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":6.5410000000000004,"5min":11.981999999999999,"15min":19.885999999999999},"last":4.2300000000000004},{"interface":"front","average":{"1min":2.867,"5min":3.4430000000000001,"15min":2.9830000000000001},"min":{"1min":0.22900000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.888999999999999,"5min":14.66,"15min":24.385000000000002},"last":4.2800000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":3.194,"5min":3.7719999999999998,"15min":3.274},"min":{"1min":0.33000000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.9760000000000009,"5min":25.84,"15min":25.84},"last":4.5030000000000001},{"interface":"front","average":{"1min":2.1509999999999998,"5min":2.77,"15min":2.7949999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.5499999999999998,"5min":25.120999999999999,"15min":25.120999999999999},"last":4.2640000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5832520,"kb_used_data":3956800,"kb_used_omap":582,"kb_used_meta":1875129,"kb_avail":88539320,"statfs":{"total":96636764160,"available":90664263680,"internally_reserved":0,"allocated":4051763200,"data_stored":18159630284,"data_compressed":12407774,"data_compressed_allocated":561299456,"data_compressed_original":14196183040,"omap_allocated":596311,"internal_metadata":1920132777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,5,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.5110000000000001,"5min":2.5939999999999999,"15min":2.5699999999999998},"min":{"1min":0.24399999999999999,"5min":0.161,"15min":0.14699999999999999},"max":{"1min":7.2779999999999996,"5min":13.217000000000001,"15min":14.496},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.1240000000000001,"5min":3.4039999999999999,"15min":3.4430000000000001},"min":{"1min":0.28299999999999997,"5min":0.192,"15min":0.188},"max":{"1min":12.435,"5min":14.779,"15min":25.012},"last":0.20300000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.6909999999999998,"5min":3.5059999999999998,"15min":3.1549999999999998},"min":{"1min":0.28199999999999997,"5min":0.19,"15min":0.19},"max":{"1min":11.538,"5min":20.216999999999999,"15min":23.443999999999999},"last":18.414000000000001},{"interface":"front","average":{"1min":1.677,"5min":1.5940000000000001,"15min":2.0430000000000001},"min":{"1min":0.255,"5min":0.193,"15min":0.187},"max":{"1min":8.7560000000000002,"5min":10.983000000000001,"15min":20.565000000000001},"last":18.353000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4053647360,"data_stored":18158784512,"data_compressed":12382219,"data_compressed_allocated":561303552,"data_compressed_original":14192607232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4050792448,"data_stored":18158706688,"data_compressed":12405009,"data_compressed_allocated":561266688,"data_compressed_original":14195191808,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4054175744,"data_stored":18158817280,"data_compressed":11796894,"data_compressed_allocated":561311744,"data_compressed_original":14192144384,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:18:43.758 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T08:18:44.388 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:18:44.389 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":93,"fsid":"f962b619-b16f-4c4b-b0ad-bbfa0665eb75","created":"2023-09-23T07:11:47.760218+0000","modified":"2023-09-23T08:18:35.410244+0000","last_up_change":"2023-09-23T07:11:56.672362+0000","last_in_change":"2023-09-23T07:11:48.725951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T07:11:57.892456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T07:12:03.106437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-09-23T07:12:12.744833+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":64,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bc742385-5f88-44ba-9305-a3dbe250ea5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6801","nonce":1002728916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6803","nonce":1002728916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6807","nonce":1002728916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":1002728916},{"type":"v1","addr":"172.21.15.140:6805","nonce":1002728916}]},"public_addr":"172.21.15.140:6801/1002728916","cluster_addr":"172.21.15.140:6803/1002728916","heartbeat_back_addr":"172.21.15.140:6807/1002728916","heartbeat_front_addr":"172.21.15.140:6805/1002728916","state":["exists","up"]},{"osd":1,"uuid":"3ee30df5-51cc-47c4-b957-eba695b78d36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6816","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6817","nonce":155684008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6819","nonce":155684008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6823","nonce":155684008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6820","nonce":155684008},{"type":"v1","addr":"172.21.15.140:6821","nonce":155684008}]},"public_addr":"172.21.15.140:6817/155684008","cluster_addr":"172.21.15.140:6819/155684008","heartbeat_back_addr":"172.21.15.140:6823/155684008","heartbeat_front_addr":"172.21.15.140:6821/155684008","state":["exists","up"]},{"osd":2,"uuid":"da2f6412-e3fc-497b-8e52-4d4661cf03c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6809","nonce":4141700485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6811","nonce":4141700485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6815","nonce":4141700485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6812","nonce":4141700485},{"type":"v1","addr":"172.21.15.140:6813","nonce":4141700485}]},"public_addr":"172.21.15.140:6809/4141700485","cluster_addr":"172.21.15.140:6811/4141700485","heartbeat_back_addr":"172.21.15.140:6815/4141700485","heartbeat_front_addr":"172.21.15.140:6813/4141700485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.943415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.880737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T07:11:53.948563+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.140:0/1974081988":"2023-09-23T08:35:01.196672+0000","172.21.15.140:0/242711256":"2023-09-23T08:32:42.106351+0000","172.21.15.140:0/947471084":"2023-09-23T08:19:44.027995+0000","172.21.15.140:0/3372138304":"2023-09-23T08:22:03.718738+0000","172.21.15.140:0/1093616448":"2023-09-23T08:25:44.576844+0000","172.21.15.140:0/3910729918":"2023-09-23T08:23:24.548674+0000","172.21.15.140:0/4167742291":"2023-09-23T08:29:13.317598+0000","172.21.15.140:0/2646156005":"2023-09-23T08:28:03.965156+0000","172.21.15.140:0/4072506271":"2023-09-23T08:20:54.328227+0000","172.21.15.140:0/2709018820":"2023-09-23T08:24:34.994843+0000","172.21.15.140:0/601208279":"2023-09-23T08:33:51.793203+0000","172.21.15.140:0/2763363184":"2023-09-23T08:26:53.791529+0000","172.21.15.140:0/771342127":"2023-09-23T08:30:22.505928+0000","172.21.15.140:0/418208292":"2023-09-23T08:31:31.991946+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T08:18:45.442 INFO:tasks.ceph:Scrubbing osd.0 2023-09-23T08:18:45.442 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-23T08:18:46.018 INFO:teuthology.orchestra.run.smithi140.stdout:{ 2023-09-23T08:18:46.018 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T08:18:46.018 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-09-23T08:18:46.057 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-23T08:18:46.815 INFO:teuthology.orchestra.run.smithi140.stderr:instructed osd(s) 0 to deep-scrub 2023-09-23T08:18:46.829 INFO:tasks.ceph:Scrubbing osd.1 2023-09-23T08:18:46.830 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-23T08:18:47.163 INFO:teuthology.orchestra.run.smithi140.stdout:{ 2023-09-23T08:18:47.164 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T08:18:47.164 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-09-23T08:18:47.200 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-23T08:18:47.844 INFO:teuthology.orchestra.run.smithi140.stderr:instructed osd(s) 1 to deep-scrub 2023-09-23T08:18:47.862 INFO:tasks.ceph:Scrubbing osd.2 2023-09-23T08:18:47.863 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-23T08:18:48.252 INFO:teuthology.orchestra.run.smithi140.stdout:{ 2023-09-23T08:18:48.253 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T08:18:48.253 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-09-23T08:18:48.272 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-23T08:18:48.949 INFO:teuthology.orchestra.run.smithi140.stderr:instructed osd(s) 2 to deep-scrub 2023-09-23T08:18:48.969 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:18:49.589 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:18:49.650 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:18:49.662 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2095,"stamp":"2023-09-23T08:18:48.929414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19951703644,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1079965,"num_read_kb":3982400,"num_write":1005812,"num_write_kb":3965413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75464,"ondisk_log_size":75464,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17335472,"kb_used_data":11888980,"kb_used_omap":1733,"kb_used_meta":5444730,"kb_avail":265780048,"statfs":{"total":289910292480,"available":272158769152,"internally_reserved":0,"allocated":12174315520,"data_stored":54478983451,"data_compressed":36590718,"data_compressed_allocated":1684029440,"data_compressed_original":42570108928,"omap_allocated":1775190,"internal_metadata":5575403946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,5,18,9,15,8,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11776000,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3241,"num_read_kb":12912,"num_write":3254,"num_write_kb":13016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015418"},"pg_stats":[{"pgid":"2.7","version":"93'5286","reported_seq":8872,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.928187+0000","last_change":"2023-09-23T07:12:04.895785+0000","last_active":"2023-09-23T08:18:45.928187+0000","last_peered":"2023-09-23T08:18:45.928187+0000","last_clean":"2023-09-23T08:18:45.928187+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:18:45.928187+0000","last_undegraded":"2023-09-23T08:18:45.928187+0000","last_fullsized":"2023-09-23T08:18:45.928187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7755","reported_seq":13402,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.927469+0000","last_change":"2023-09-23T07:12:04.899539+0000","last_active":"2023-09-23T08:18:45.927469+0000","last_peered":"2023-09-23T08:18:45.927469+0000","last_clean":"2023-09-23T08:18:45.927469+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:18:45.927469+0000","last_undegraded":"2023-09-23T08:18:45.927469+0000","last_fullsized":"2023-09-23T08:18:45.927469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3767","reported_seq":6167,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.927897+0000","last_change":"2023-09-23T07:12:04.895694+0000","last_active":"2023-09-23T08:18:45.927897+0000","last_peered":"2023-09-23T08:18:45.927897+0000","last_clean":"2023-09-23T08:18:45.927897+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:18:45.927897+0000","last_undegraded":"2023-09-23T08:18:45.927897+0000","last_fullsized":"2023-09-23T08:18:45.927897+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'253278","reported_seq":507268,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:47.095529+0000","last_change":"2023-09-23T07:35:03.039194+0000","last_active":"2023-09-23T08:18:47.095529+0000","last_peered":"2023-09-23T08:18:47.095529+0000","last_clean":"2023-09-23T08:18:47.095529+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:18:47.095529+0000","last_undegraded":"2023-09-23T08:18:47.095529+0000","last_fullsized":"2023-09-23T08:18:47.095529+0000","mapping_epoch":18,"log_start":"90'243178","ondisk_log_start":"90'243178","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5099274240,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":255190,"num_read_kb":999148,"num_write":251896,"num_write_kb":1007584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"88'509","reported_seq":2901,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.931275+0000","last_change":"2023-09-23T07:12:04.898519+0000","last_active":"2023-09-23T08:18:45.931275+0000","last_peered":"2023-09-23T08:18:45.931275+0000","last_clean":"2023-09-23T08:18:45.931275+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:18:45.931275+0000","last_undegraded":"2023-09-23T08:18:45.931275+0000","last_fullsized":"2023-09-23T08:18:45.931275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:26:43.861575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'239522","reported_seq":480250,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:48.093126+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:18:48.093126+0000","last_peered":"2023-09-23T08:18:48.093126+0000","last_clean":"2023-09-23T08:18:48.093126+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:18:48.093126+0000","last_undegraded":"2023-09-23T08:18:48.093126+0000","last_fullsized":"2023-09-23T08:18:48.093126+0000","mapping_epoch":18,"log_start":"90'229426","ondisk_log_start":"90'229426","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4832198675,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":241850,"num_read_kb":947340,"num_write":238221,"num_write_kb":952878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":251,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:35.433368+0000","last_change":"2023-09-23T07:12:00.034973+0000","last_active":"2023-09-23T08:18:35.433368+0000","last_peered":"2023-09-23T08:18:35.433368+0000","last_clean":"2023-09-23T08:18:35.433368+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:35.433368+0000","last_undegraded":"2023-09-23T08:18:35.433368+0000","last_fullsized":"2023-09-23T08:18:35.433368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:11:58.862445+0000","last_clean_scrub_stamp":"2023-09-23T07:11:58.862445+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'6446","reported_seq":9974,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.927593+0000","last_change":"2023-09-23T07:12:04.898220+0000","last_active":"2023-09-23T08:18:45.927593+0000","last_peered":"2023-09-23T08:18:45.927593+0000","last_clean":"2023-09-23T08:18:45.927593+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:18:45.927593+0000","last_undegraded":"2023-09-23T08:18:45.927593+0000","last_fullsized":"2023-09-23T08:18:45.927593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'241223","reported_seq":483614,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:48.083278+0000","last_change":"2023-09-23T07:35:03.043707+0000","last_active":"2023-09-23T08:18:48.083278+0000","last_peered":"2023-09-23T08:18:48.083278+0000","last_clean":"2023-09-23T08:18:48.083278+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:18:48.083278+0000","last_undegraded":"2023-09-23T08:18:48.083278+0000","last_fullsized":"2023-09-23T08:18:48.083278+0000","mapping_epoch":18,"log_start":"90'231131","ondisk_log_start":"90'231131","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4850429952,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":243507,"num_read_kb":953808,"num_write":239931,"num_write_kb":959724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'2310","reported_seq":3581,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.929231+0000","last_change":"2023-09-23T07:12:04.898441+0000","last_active":"2023-09-23T08:18:45.929231+0000","last_peered":"2023-09-23T08:18:45.929231+0000","last_clean":"2023-09-23T08:18:45.929231+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:18:45.929231+0000","last_undegraded":"2023-09-23T08:18:45.929231+0000","last_fullsized":"2023-09-23T08:18:45.929231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T18:23:50.018321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'257423","reported_seq":515128,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:47.094358+0000","last_change":"2023-09-23T07:35:03.045902+0000","last_active":"2023-09-23T08:18:47.094358+0000","last_peered":"2023-09-23T08:18:47.094358+0000","last_clean":"2023-09-23T08:18:47.094358+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:18:47.094358+0000","last_undegraded":"2023-09-23T08:18:47.094358+0000","last_fullsized":"2023-09-23T08:18:47.094358+0000","mapping_epoch":18,"log_start":"90'247380","ondisk_log_start":"90'247380","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5169201152,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":258882,"num_read_kb":1014340,"num_write":256066,"num_write_kb":1024264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4511","reported_seq":6891,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.930323+0000","last_change":"2023-09-23T07:12:04.900345+0000","last_active":"2023-09-23T08:18:45.930323+0000","last_peered":"2023-09-23T08:18:45.930323+0000","last_clean":"2023-09-23T08:18:45.930323+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:18:45.930323+0000","last_undegraded":"2023-09-23T08:18:45.930323+0000","last_fullsized":"2023-09-23T08:18:45.930323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'4452","reported_seq":8565,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:45.927636+0000","last_change":"2023-09-23T07:12:04.898103+0000","last_active":"2023-09-23T08:18:45.927636+0000","last_peered":"2023-09-23T08:18:45.927636+0000","last_clean":"2023-09-23T08:18:45.927636+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:18:45.927636+0000","last_undegraded":"2023-09-23T08:18:45.927636+0000","last_fullsized":"2023-09-23T08:18:45.927636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:03.875992+0000","last_clean_scrub_stamp":"2023-09-23T07:12:03.875992+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19951104019,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":999429,"num_read_kb":3914636,"num_write":986114,"num_write_kb":3944450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12172169216,"data_stored":54476693504,"data_compressed":36584976,"data_compressed_allocated":1683951616,"data_compressed_original":42567385088,"omap_allocated":0,"internal_metadata":0},"log_size":40331,"ondisk_log_size":40331,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35036,"ondisk_log_size":35036,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5698336,"kb_used_data":3963028,"kb_used_omap":572,"kb_used_meta":1734723,"kb_avail":88673504,"statfs":{"total":96636764160,"available":90801668096,"internally_reserved":0,"allocated":4058140672,"data_stored":18159268609,"data_compressed":11796816,"data_compressed_allocated":561319936,"data_compressed_original":14189461504,"omap_allocated":585792,"internal_metadata":1776357312},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":2.0110000000000001,"15min":2.1899999999999999},"min":{"1min":0.27300000000000002,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.0830000000000002,"5min":13.019,"15min":16.388999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.284,"5min":2.2799999999999998,"15min":2.5390000000000001},"min":{"1min":0.29899999999999999,"5min":0.219,"15min":0.185},"max":{"1min":5.1109999999999998,"5min":22.266999999999999,"15min":22.266999999999999},"last":0.379}]},{"osd":1,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":3.165,"5min":3.456,"15min":3.2200000000000002},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.186},"max":{"1min":11.971,"5min":19.433,"15min":23.045999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":2.7130000000000001,"5min":2.827,"15min":3.048},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.92,"5min":11.92,"15min":23.100999999999999},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359739167,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5788232,"kb_used_data":3962496,"kb_used_omap":579,"kb_used_meta":1825148,"kb_avail":88583608,"statfs":{"total":96636764160,"available":90709614592,"internally_reserved":0,"allocated":4057595904,"data_stored":18159855182,"data_compressed":12385661,"data_compressed_allocated":561356800,"data_compressed_original":14190682112,"omap_allocated":593087,"internal_metadata":1868952385},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":2.4980000000000002,"15min":2.7229999999999999},"min":{"1min":0.24099999999999999,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":6.5410000000000004,"5min":11.981999999999999,"15min":19.885999999999999},"last":5.8879999999999999},{"interface":"front","average":{"1min":2.867,"5min":3.4430000000000001,"15min":2.9830000000000001},"min":{"1min":0.22900000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.888999999999999,"5min":14.66,"15min":24.385000000000002},"last":0.60899999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:18:03 2023","interfaces":[{"interface":"back","average":{"1min":3.194,"5min":3.7719999999999998,"15min":3.274},"min":{"1min":0.33000000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.9760000000000009,"5min":25.84,"15min":25.84},"last":2.7320000000000002},{"interface":"front","average":{"1min":2.1509999999999998,"5min":2.77,"15min":2.7949999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.5499999999999998,"5min":25.120999999999999,"15min":25.120999999999999},"last":13.435}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5848904,"kb_used_data":3963456,"kb_used_omap":582,"kb_used_meta":1884857,"kb_avail":88522936,"statfs":{"total":96636764160,"available":90647486464,"internally_reserved":0,"allocated":4058578944,"data_stored":18159859660,"data_compressed":12408241,"data_compressed_allocated":561352704,"data_compressed_original":14189965312,"omap_allocated":596311,"internal_metadata":1930094249},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,0,1,11,2,7,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.5110000000000001,"5min":2.5939999999999999,"15min":2.5699999999999998},"min":{"1min":0.24399999999999999,"5min":0.161,"15min":0.14699999999999999},"max":{"1min":7.2779999999999996,"5min":13.217000000000001,"15min":14.496},"last":0.17999999999999999},{"interface":"front","average":{"1min":3.1240000000000001,"5min":3.4039999999999999,"15min":3.4430000000000001},"min":{"1min":0.28299999999999997,"5min":0.192,"15min":0.188},"max":{"1min":12.435,"5min":14.779,"15min":25.012},"last":1.7789999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.6909999999999998,"5min":3.5059999999999998,"15min":3.1549999999999998},"min":{"1min":0.28199999999999997,"5min":0.19,"15min":0.19},"max":{"1min":11.538,"5min":20.216999999999999,"15min":23.443999999999999},"last":5.9740000000000002},{"interface":"front","average":{"1min":1.677,"5min":1.5940000000000001,"15min":2.0430000000000001},"min":{"1min":0.255,"5min":0.193,"15min":0.187},"max":{"1min":8.7560000000000002,"5min":10.983000000000001,"15min":20.565000000000001},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4057038848,"data_stored":18158882816,"data_compressed":12382596,"data_compressed_allocated":561319936,"data_compressed_original":14189449216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4057616384,"data_stored":18158931968,"data_compressed":12405643,"data_compressed_allocated":561315840,"data_compressed_original":14188945408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4057513984,"data_stored":18158878720,"data_compressed":11796737,"data_compressed_allocated":561315840,"data_compressed_original":14188990464,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:18:49.667 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.667 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.667 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.668 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.668 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.668 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.669 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-23T07:11:58.862445+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=11, tm_sec=58, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.669 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.669 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.670 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.670 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.671 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.671 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-23T07:12:03.875992+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:18:49.671 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:19:09.672 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:19:10.265 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:19:10.275 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:19:10.294 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2105,"stamp":"2023-09-23T08:19:08.951384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19966776924,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1083993,"num_read_kb":3998460,"num_write":1010001,"num_write_kb":3982169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75521,"ondisk_log_size":75521,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16840024,"kb_used_data":11928624,"kb_used_omap":1733,"kb_used_meta":4909626,"kb_avail":266275496,"statfs":{"total":289910292480,"available":272666107904,"internally_reserved":0,"allocated":12214910976,"data_stored":54479892763,"data_compressed":36587251,"data_compressed_allocated":1684033536,"data_compressed_original":42532933632,"omap_allocated":1775190,"internal_metadata":5027457450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,8,11,25,16,14,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11272192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3015,"num_read_kb":12016,"num_write":3132,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11012,"num_omap_keys":254,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009337"},"pg_stats":[{"pgid":"2.7","version":"93'5314","reported_seq":8920,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.407619+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:19:06.407619+0000","last_peered":"2023-09-23T08:19:06.407619+0000","last_clean":"2023-09-23T08:19:06.407619+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:19:06.407619+0000","last_undegraded":"2023-09-23T08:19:06.407619+0000","last_fullsized":"2023-09-23T08:19:06.407619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7792","reported_seq":13463,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.409525+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:19:06.409525+0000","last_peered":"2023-09-23T08:19:06.409525+0000","last_clean":"2023-09-23T08:19:06.409525+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:19:06.409525+0000","last_undegraded":"2023-09-23T08:19:06.409525+0000","last_fullsized":"2023-09-23T08:19:06.409525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3788","reported_seq":6204,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.407575+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:19:06.407575+0000","last_peered":"2023-09-23T08:19:06.407575+0000","last_clean":"2023-09-23T08:19:06.407575+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:19:06.407575+0000","last_undegraded":"2023-09-23T08:19:06.407575+0000","last_fullsized":"2023-09-23T08:19:06.407575+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3788,"log_dups_size":0,"ondisk_log_size":3788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'254342","reported_seq":509409,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:07.508927+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:19:07.508927+0000","last_peered":"2023-09-23T08:19:07.508927+0000","last_clean":"2023-09-23T08:19:07.508927+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:19:07.508927+0000","last_undegraded":"2023-09-23T08:19:07.508927+0000","last_fullsized":"2023-09-23T08:19:07.508927+0000","mapping_epoch":18,"log_start":"90'244292","ondisk_log_start":"90'244292","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":74,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5103091712,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":256258,"num_read_kb":1003408,"num_write":252960,"num_write_kb":1011840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2939,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.419605+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:19:06.419605+0000","last_peered":"2023-09-23T08:19:06.419605+0000","last_clean":"2023-09-23T08:19:06.419605+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:19:06.419605+0000","last_undegraded":"2023-09-23T08:19:06.419605+0000","last_fullsized":"2023-09-23T08:19:06.419605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'240518","reported_seq":482214,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:08.492711+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:19:08.492711+0000","last_peered":"2023-09-23T08:19:08.492711+0000","last_clean":"2023-09-23T08:19:08.492711+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:19:08.492711+0000","last_undegraded":"2023-09-23T08:19:08.492711+0000","last_fullsized":"2023-09-23T08:19:08.492711+0000","mapping_epoch":18,"log_start":"90'230438","ondisk_log_start":"90'230438","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4835745811,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":242816,"num_read_kb":951188,"num_write":239217,"num_write_kb":956862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":262,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:51.444335+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:51.444335+0000","last_peered":"2023-09-23T08:18:51.444335+0000","last_clean":"2023-09-23T08:18:51.444335+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:51.444335+0000","last_undegraded":"2023-09-23T08:18:51.444335+0000","last_fullsized":"2023-09-23T08:18:51.444335+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'6473","reported_seq":10021,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.409799+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:19:06.409799+0000","last_peered":"2023-09-23T08:19:06.409799+0000","last_clean":"2023-09-23T08:19:06.409799+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:19:06.409799+0000","last_undegraded":"2023-09-23T08:19:06.409799+0000","last_fullsized":"2023-09-23T08:19:06.409799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'242250","reported_seq":485590,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:08.502923+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:19:08.502923+0000","last_peered":"2023-09-23T08:19:08.502923+0000","last_clean":"2023-09-23T08:19:08.502923+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:19:08.502923+0000","last_undegraded":"2023-09-23T08:19:08.502923+0000","last_fullsized":"2023-09-23T08:19:08.502923+0000","mapping_epoch":18,"log_start":"90'232245","ondisk_log_start":"90'232245","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":4,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4854128640,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":244449,"num_read_kb":957556,"num_write":240958,"num_write_kb":963832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3620,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:01.312820+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:19:01.312820+0000","last_peered":"2023-09-23T08:19:01.312820+0000","last_clean":"2023-09-23T08:19:01.312820+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:19:01.312820+0000","last_undegraded":"2023-09-23T08:19:01.312820+0000","last_fullsized":"2023-09-23T08:19:01.312820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'258525","reported_seq":517290,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:07.526752+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:19:07.526752+0000","last_peered":"2023-09-23T08:19:07.526752+0000","last_clean":"2023-09-23T08:19:07.526752+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:19:07.526752+0000","last_undegraded":"2023-09-23T08:19:07.526752+0000","last_fullsized":"2023-09-23T08:19:07.526752+0000","mapping_epoch":18,"log_start":"90'248494","ondisk_log_start":"90'248494","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":83,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5173211136,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":259934,"num_read_kb":1018544,"num_write":257168,"num_write_kb":1028672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4547","reported_seq":6943,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.800645+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:19:06.800645+0000","last_peered":"2023-09-23T08:19:06.800645+0000","last_clean":"2023-09-23T08:19:06.800645+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:19:06.800645+0000","last_undegraded":"2023-09-23T08:19:06.800645+0000","last_fullsized":"2023-09-23T08:19:06.800645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'4477","reported_seq":8610,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:06.407786+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:19:06.407786+0000","last_peered":"2023-09-23T08:19:06.407786+0000","last_clean":"2023-09-23T08:19:06.407786+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:19:06.407786+0000","last_undegraded":"2023-09-23T08:19:06.407786+0000","last_fullsized":"2023-09-23T08:19:06.407786+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19966177299,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1003457,"num_read_kb":3930696,"num_write":990303,"num_write_kb":3961206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12211818496,"data_stored":54477578240,"data_compressed":36580743,"data_compressed_allocated":1683959808,"data_compressed_original":42531106816,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35256,"ondisk_log_size":35256,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739174,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5797708,"kb_used_data":3976320,"kb_used_omap":572,"kb_used_meta":1820803,"kb_avail":88574132,"statfs":{"total":96636764160,"available":90699911168,"internally_reserved":0,"allocated":4071751680,"data_stored":18159579905,"data_compressed":11795503,"data_compressed_allocated":561319936,"data_compressed_original":14177013760,"omap_allocated":585792,"internal_metadata":1864503232},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":2.0110000000000001,"15min":2.1899999999999999},"min":{"1min":0.27300000000000002,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.0830000000000002,"5min":13.019,"15min":16.388999999999999},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.284,"5min":2.2799999999999998,"15min":2.5390000000000001},"min":{"1min":0.29899999999999999,"5min":0.219,"15min":0.185},"max":{"1min":5.1109999999999998,"5min":22.266999999999999,"15min":22.266999999999999},"last":2.0190000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:18:22 2023","interfaces":[{"interface":"back","average":{"1min":3.165,"5min":3.456,"15min":3.2200000000000002},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.186},"max":{"1min":11.971,"5min":19.433,"15min":23.045999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.7130000000000001,"5min":2.827,"15min":3.048},"min":{"1min":0.24099999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.92,"5min":11.92,"15min":23.100999999999999},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5887196,"kb_used_data":3975444,"kb_used_omap":579,"kb_used_meta":1911164,"kb_avail":88484644,"statfs":{"total":96636764160,"available":90608275456,"internally_reserved":0,"allocated":4070854656,"data_stored":18160158286,"data_compressed":12385115,"data_compressed_allocated":561356800,"data_compressed_original":14178521088,"omap_allocated":593087,"internal_metadata":1957032769},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,2,16,8,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":2.7799999999999998,"15min":2.7469999999999999},"min":{"1min":0.26200000000000001,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":10.904999999999999,"5min":11.981999999999999,"15min":19.885999999999999},"last":0.496},{"interface":"front","average":{"1min":2.4990000000000001,"5min":3.351,"15min":2.9390000000000001},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":6.1429999999999998,"5min":14.66,"15min":24.385000000000002},"last":0.22800000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.4329999999999998,"5min":3.383,"15min":3.3410000000000002},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.15,"5min":25.84,"15min":25.84},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.5990000000000002,"5min":2.9710000000000001,"15min":2.7890000000000001},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.20699999999999999},"max":{"1min":13.435,"5min":25.120999999999999,"15min":25.120999999999999},"last":0.53400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739171,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5155120,"kb_used_data":3976860,"kb_used_omap":582,"kb_used_meta":1177657,"kb_avail":89216720,"statfs":{"total":96636764160,"available":91357921280,"internally_reserved":0,"allocated":4072304640,"data_stored":18160154572,"data_compressed":12406633,"data_compressed_allocated":561356800,"data_compressed_original":14177398784,"omap_allocated":596311,"internal_metadata":1205921449},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,2,5,4,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.5110000000000001,"5min":2.5939999999999999,"15min":2.5699999999999998},"min":{"1min":0.24399999999999999,"5min":0.161,"15min":0.14699999999999999},"max":{"1min":7.2779999999999996,"5min":13.217000000000001,"15min":14.496},"last":3.5499999999999998},{"interface":"front","average":{"1min":3.1240000000000001,"5min":3.4039999999999999,"15min":3.4430000000000001},"min":{"1min":0.28299999999999997,"5min":0.192,"15min":0.188},"max":{"1min":12.435,"5min":14.779,"15min":25.012},"last":5.6959999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:18:14 2023","interfaces":[{"interface":"back","average":{"1min":2.6909999999999998,"5min":3.5059999999999998,"15min":3.1549999999999998},"min":{"1min":0.28199999999999997,"5min":0.19,"15min":0.19},"max":{"1min":11.538,"5min":20.216999999999999,"15min":23.443999999999999},"last":5.5110000000000001},{"interface":"front","average":{"1min":1.677,"5min":1.5940000000000001,"15min":2.0430000000000001},"min":{"1min":0.255,"5min":0.193,"15min":0.187},"max":{"1min":8.7560000000000002,"5min":10.983000000000001,"15min":20.565000000000001},"last":0.28499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4070236160,"data_stored":18159181824,"data_compressed":12381774,"data_compressed_allocated":561319936,"data_compressed_original":14177370112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4070891520,"data_stored":18159194112,"data_compressed":12403646,"data_compressed_allocated":561319936,"data_compressed_original":14176772096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4070690816,"data_stored":18159202304,"data_compressed":11795323,"data_compressed_allocated":561319936,"data_compressed_original":14176964608,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:19:10.295 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:10.296 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:10.296 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:10.296 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:10.296 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:19:13.628 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:19:13.628 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:19:13.629 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:19:13.630 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:19:13.630 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:19:13.630 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:19:13.630 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:19:13.630 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:13.626+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:19:28.791 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:19:28.791 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:19:28.791 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:19:28.792 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:19:28.793 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:19:28.793 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:19:28.793 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:19:28.793 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:19:28.794 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:19:28.787+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:19:30.297 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:19:30.969 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:19:30.991 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:19:31.000 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2115,"stamp":"2023-09-23T08:19:28.964669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19981350492,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1087777,"num_read_kb":4013568,"num_write":1013971,"num_write_kb":3998049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75767,"ondisk_log_size":75767,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14955028,"kb_used_data":11964412,"kb_used_omap":1735,"kb_used_meta":2988856,"kb_avail":268160492,"statfs":{"total":289910292480,"available":274596343808,"internally_reserved":0,"allocated":12251557888,"data_stored":54481395995,"data_compressed":36588177,"data_compressed_allocated":1684176896,"data_compressed_original":42500190208,"omap_allocated":1776870,"internal_metadata":3060589338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,0,6,8,16,12,19,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10797056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2853,"num_read_kb":11396,"num_write":2964,"num_write_kb":11856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010438"},"pg_stats":[{"pgid":"2.7","version":"93'5342","reported_seq":8958,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:26.894815+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:19:26.894815+0000","last_peered":"2023-09-23T08:19:26.894815+0000","last_clean":"2023-09-23T08:19:26.894815+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:19:26.894815+0000","last_undegraded":"2023-09-23T08:19:26.894815+0000","last_fullsized":"2023-09-23T08:19:26.894815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7829","reported_seq":13516,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:28.774127+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:19:28.774127+0000","last_peered":"2023-09-23T08:19:28.774127+0000","last_clean":"2023-09-23T08:19:28.774127+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:19:28.774127+0000","last_undegraded":"2023-09-23T08:19:28.774127+0000","last_fullsized":"2023-09-23T08:19:28.774127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3807","reported_seq":6231,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:28.773538+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:19:28.773538+0000","last_peered":"2023-09-23T08:19:28.773538+0000","last_clean":"2023-09-23T08:19:28.773538+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:19:28.773538+0000","last_undegraded":"2023-09-23T08:19:28.773538+0000","last_fullsized":"2023-09-23T08:19:28.773538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"93'255415","reported_seq":511427,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:28.183239+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:19:28.183239+0000","last_peered":"2023-09-23T08:19:28.183239+0000","last_clean":"2023-09-23T08:19:28.183239+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:19:28.183239+0000","last_undegraded":"2023-09-23T08:19:28.183239+0000","last_fullsized":"2023-09-23T08:19:28.183239+0000","mapping_epoch":18,"log_start":"91'245320","ondisk_log_start":"91'245320","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":171,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5107044352,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":257191,"num_read_kb":1007128,"num_write":254033,"num_write_kb":1016132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2947,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:26.889088+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:19:26.889088+0000","last_peered":"2023-09-23T08:19:26.889088+0000","last_clean":"2023-09-23T08:19:26.889088+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:19:26.889088+0000","last_undegraded":"2023-09-23T08:19:26.889088+0000","last_fullsized":"2023-09-23T08:19:26.889088+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'241461","reported_seq":484079,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:28.785651+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:19:28.785651+0000","last_peered":"2023-09-23T08:19:28.785651+0000","last_clean":"2023-09-23T08:19:28.785651+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:19:28.785651+0000","last_undegraded":"2023-09-23T08:19:28.785651+0000","last_fullsized":"2023-09-23T08:19:28.785651+0000","mapping_epoch":18,"log_start":"91'231455","ondisk_log_start":"91'231455","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4839198739,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":243739,"num_read_kb":954876,"num_write":240160,"num_write_kb":960634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":262,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:51.444335+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:51.444335+0000","last_peered":"2023-09-23T08:18:51.444335+0000","last_clean":"2023-09-23T08:18:51.444335+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:51.444335+0000","last_undegraded":"2023-09-23T08:18:51.444335+0000","last_fullsized":"2023-09-23T08:18:51.444335+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'6500","reported_seq":10060,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:28.782900+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:19:28.782900+0000","last_peered":"2023-09-23T08:19:28.782900+0000","last_clean":"2023-09-23T08:19:28.782900+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:19:28.782900+0000","last_undegraded":"2023-09-23T08:19:28.782900+0000","last_fullsized":"2023-09-23T08:19:28.782900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'243136","reported_seq":487473,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:28.731319+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:19:28.731319+0000","last_peered":"2023-09-23T08:19:28.731319+0000","last_clean":"2023-09-23T08:19:28.731319+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:19:28.731319+0000","last_undegraded":"2023-09-23T08:19:28.731319+0000","last_fullsized":"2023-09-23T08:19:28.731319+0000","mapping_epoch":18,"log_start":"90'233059","ondisk_log_start":"90'233059","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":200,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 22s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4857430016,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":245426,"num_read_kb":961456,"num_write":241844,"num_write_kb":967376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3624,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:21.768430+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:19:21.768430+0000","last_peered":"2023-09-23T08:19:21.768430+0000","last_clean":"2023-09-23T08:19:21.768430+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:19:21.768430+0000","last_undegraded":"2023-09-23T08:19:21.768430+0000","last_fullsized":"2023-09-23T08:19:21.768430+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"93'259593","reported_seq":519321,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:28.233688+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:19:28.233688+0000","last_peered":"2023-09-23T08:19:28.233688+0000","last_clean":"2023-09-23T08:19:28.233688+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:19:28.233688+0000","last_undegraded":"2023-09-23T08:19:28.233688+0000","last_fullsized":"2023-09-23T08:19:28.233688+0000","mapping_epoch":18,"log_start":"91'249511","ondisk_log_start":"91'249511","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":190,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 35s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5177077760,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":260885,"num_read_kb":1022344,"num_write":258236,"num_write_kb":1032944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4561","reported_seq":6966,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:26.890583+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:19:26.890583+0000","last_peered":"2023-09-23T08:19:26.890583+0000","last_clean":"2023-09-23T08:19:26.890583+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:19:26.890583+0000","last_undegraded":"2023-09-23T08:19:26.890583+0000","last_fullsized":"2023-09-23T08:19:26.890583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'4504","reported_seq":8648,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:19:28.783588+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:19:28.783588+0000","last_peered":"2023-09-23T08:19:28.783588+0000","last_clean":"2023-09-23T08:19:28.783588+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:19:28.783588+0000","last_undegraded":"2023-09-23T08:19:28.783588+0000","last_fullsized":"2023-09-23T08:19:28.783588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19980750867,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1007241,"num_read_kb":3945804,"num_write":994273,"num_write_kb":3977086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12248530944,"data_stored":54479077376,"data_compressed":36582490,"data_compressed_allocated":1684107264,"data_compressed_original":42498293760,"omap_allocated":0,"internal_metadata":0},"log_size":40260,"ondisk_log_size":40260,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35408,"ondisk_log_size":35408,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5085640,"kb_used_data":3988096,"kb_used_omap":572,"kb_used_meta":1096963,"kb_avail":89286200,"statfs":{"total":96636764160,"available":91429068800,"internally_reserved":0,"allocated":4083810304,"data_stored":18160055041,"data_compressed":11795932,"data_compressed_allocated":561369088,"data_compressed_original":14166220800,"omap_allocated":585792,"internal_metadata":1123291072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":2.056,"15min":2.1360000000000001},"min":{"1min":0.17999999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":11.561,"5min":13.019,"15min":16.388999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.9039999999999999,"5min":2.1459999999999999,"15min":2.4750000000000001},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":11.597,"5min":22.266999999999999,"15min":22.266999999999999},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":2.8319999999999999,"5min":3.4209999999999998,"15min":3.1419999999999999},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":15.848000000000001,"5min":19.433,"15min":23.045999999999999},"last":8.282},{"interface":"front","average":{"1min":3.6520000000000001,"5min":3.0099999999999998,"15min":3.0209999999999999},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":28.257000000000001,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.41999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739175,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4652624,"kb_used_data":3987784,"kb_used_omap":580,"kb_used_meta":664251,"kb_avail":89719216,"statfs":{"total":96636764160,"available":91872477184,"internally_reserved":0,"allocated":4083490816,"data_stored":18160662094,"data_compressed":12385306,"data_compressed_allocated":561401856,"data_compressed_original":14167220224,"omap_allocated":594767,"internal_metadata":680193201},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,0,5,11,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":2.7799999999999998,"15min":2.7469999999999999},"min":{"1min":0.26200000000000001,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":10.904999999999999,"5min":11.981999999999999,"15min":19.885999999999999},"last":0.183},{"interface":"front","average":{"1min":2.4990000000000001,"5min":3.351,"15min":2.9390000000000001},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":6.1429999999999998,"5min":14.66,"15min":24.385000000000002},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.4329999999999998,"5min":3.383,"15min":3.3410000000000002},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.15,"5min":25.84,"15min":25.84},"last":8.1839999999999993},{"interface":"front","average":{"1min":2.5990000000000002,"5min":2.9710000000000001,"15min":2.7890000000000001},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.20699999999999999},"max":{"1min":13.435,"5min":25.120999999999999,"15min":25.120999999999999},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739175,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5216764,"kb_used_data":3988532,"kb_used_omap":582,"kb_used_meta":1227641,"kb_avail":89155076,"statfs":{"total":96636764160,"available":91294797824,"internally_reserved":0,"allocated":4084256768,"data_stored":18160678860,"data_compressed":12406939,"data_compressed_allocated":561405952,"data_compressed_original":14166749184,"omap_allocated":596311,"internal_metadata":1257105065},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,13,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":2.5,"15min":2.5640000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":6.2729999999999997,"5min":13.217000000000001,"15min":14.496},"last":0.252},{"interface":"front","average":{"1min":2.794,"5min":3.1400000000000001,"15min":3.3319999999999999},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.189},"max":{"1min":11.513999999999999,"5min":14.779,"15min":25.012},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":4.5629999999999997,"5min":3.5880000000000001,"15min":3.286},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":18.414000000000001,"5min":20.216999999999999,"15min":23.443999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":2.484,"5min":1.917,"15min":2.0169999999999999},"min":{"1min":0.249,"5min":0.193,"15min":0.187},"max":{"1min":18.353000000000002,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4082671616,"data_stored":18159685632,"data_compressed":12382263,"data_compressed_allocated":561369088,"data_compressed_original":14166253568,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4083023872,"data_stored":18159706112,"data_compressed":12404253,"data_compressed_allocated":561369088,"data_compressed_original":14165938176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4082835456,"data_stored":18159685632,"data_compressed":11795974,"data_compressed_allocated":561369088,"data_compressed_original":14166102016,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:19:31.002 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:31.002 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:31.002 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:31.002 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:31.003 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:19:51.005 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:19:51.686 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:19:51.715 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:19:51.724 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2127,"stamp":"2023-09-23T08:19:50.976925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19996161628,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1091906,"num_read_kb":4030072,"num_write":1018050,"num_write_kb":4014365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75826,"ondisk_log_size":75826,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15222088,"kb_used_data":12002416,"kb_used_omap":1735,"kb_used_meta":3217912,"kb_avail":267893432,"statfs":{"total":289910292480,"available":274322874368,"internally_reserved":0,"allocated":12290473984,"data_stored":54482236665,"data_compressed":36587579,"data_compressed_allocated":1684226048,"data_compressed_original":42463928320,"omap_allocated":1776870,"internal_metadata":3295142682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,1,0,10,18,31,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7458816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2105,"num_read_kb":8412,"num_write":2040,"num_write_kb":8160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008281"},"pg_stats":[{"pgid":"2.7","version":"94'5369","reported_seq":9001,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:48.965043+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:19:48.965043+0000","last_peered":"2023-09-23T08:19:48.965043+0000","last_clean":"2023-09-23T08:19:48.965043+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:19:48.965043+0000","last_undegraded":"2023-09-23T08:19:48.965043+0000","last_fullsized":"2023-09-23T08:19:48.965043+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5369,"log_dups_size":0,"ondisk_log_size":5369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7865","reported_seq":13569,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:48.977809+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:19:48.977809+0000","last_peered":"2023-09-23T08:19:48.977809+0000","last_clean":"2023-09-23T08:19:48.977809+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:19:48.977809+0000","last_undegraded":"2023-09-23T08:19:48.977809+0000","last_fullsized":"2023-09-23T08:19:48.977809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7865,"log_dups_size":0,"ondisk_log_size":7865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3825","reported_seq":6259,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:48.949138+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:19:48.949138+0000","last_peered":"2023-09-23T08:19:48.949138+0000","last_clean":"2023-09-23T08:19:48.949138+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:19:48.949138+0000","last_undegraded":"2023-09-23T08:19:48.949138+0000","last_fullsized":"2023-09-23T08:19:48.949138+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3825,"log_dups_size":0,"ondisk_log_size":3825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"94'256480","reported_seq":513552,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:48.829078+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:19:48.829078+0000","last_peered":"2023-09-23T08:19:48.829078+0000","last_clean":"2023-09-23T08:19:48.829078+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:19:48.829078+0000","last_undegraded":"2023-09-23T08:19:48.829078+0000","last_fullsized":"2023-09-23T08:19:48.829078+0000","mapping_epoch":18,"log_start":"91'246445","ondisk_log_start":"91'246445","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":325,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 52s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5110939648,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":258241,"num_read_kb":1011324,"num_write":255098,"num_write_kb":1020392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2956,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:47.386969+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:19:47.386969+0000","last_peered":"2023-09-23T08:19:47.386969+0000","last_clean":"2023-09-23T08:19:47.386969+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:19:47.386969+0000","last_undegraded":"2023-09-23T08:19:47.386969+0000","last_fullsized":"2023-09-23T08:19:47.386969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'242427","reported_seq":486040,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:49.192404+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:19:49.192404+0000","last_peered":"2023-09-23T08:19:49.192404+0000","last_clean":"2023-09-23T08:19:49.192404+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:19:49.192404+0000","last_undegraded":"2023-09-23T08:19:49.192404+0000","last_fullsized":"2023-09-23T08:19:49.192404+0000","mapping_epoch":18,"log_start":"91'232366","ondisk_log_start":"91'232366","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4842909715,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":244733,"num_read_kb":958848,"num_write":241126,"num_write_kb":964498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":263,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:53.202923+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:53.202923+0000","last_peered":"2023-09-23T08:18:53.202923+0000","last_clean":"2023-09-23T08:18:53.202923+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:53.202923+0000","last_undegraded":"2023-09-23T08:18:53.202923+0000","last_fullsized":"2023-09-23T08:18:53.202923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'6527","reported_seq":10101,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:48.981946+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:19:48.981946+0000","last_peered":"2023-09-23T08:19:48.981946+0000","last_clean":"2023-09-23T08:19:48.981946+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:19:48.981946+0000","last_undegraded":"2023-09-23T08:19:48.981946+0000","last_fullsized":"2023-09-23T08:19:48.981946+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6527,"log_dups_size":0,"ondisk_log_size":6527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"94'244132","reported_seq":489478,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:49.176994+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:19:49.176994+0000","last_peered":"2023-09-23T08:19:49.176994+0000","last_clean":"2023-09-23T08:19:49.176994+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:19:49.176994+0000","last_undegraded":"2023-09-23T08:19:49.176994+0000","last_fullsized":"2023-09-23T08:19:49.176994+0000","mapping_epoch":18,"log_start":"91'234070","ondisk_log_start":"91'234070","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":383,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 42s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4860878848,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":246422,"num_read_kb":965436,"num_write":242840,"num_write_kb":971360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3629,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:45.483819+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:19:45.483819+0000","last_peered":"2023-09-23T08:19:45.483819+0000","last_clean":"2023-09-23T08:19:45.483819+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:19:45.483819+0000","last_undegraded":"2023-09-23T08:19:45.483819+0000","last_fullsized":"2023-09-23T08:19:45.483819+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"94'260645","reported_seq":521478,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:19:48.873447+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:19:48.873447+0000","last_peered":"2023-09-23T08:19:48.873447+0000","last_clean":"2023-09-23T08:19:48.873447+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:19:48.873447+0000","last_undegraded":"2023-09-23T08:19:48.873447+0000","last_fullsized":"2023-09-23T08:19:48.873447+0000","mapping_epoch":18,"log_start":"91'250631","ondisk_log_start":"91'250631","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":312,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5180833792,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":261974,"num_read_kb":1026700,"num_write":259288,"num_write_kb":1037152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'4573","reported_seq":6987,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:47.367093+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:19:47.367093+0000","last_peered":"2023-09-23T08:19:47.367093+0000","last_clean":"2023-09-23T08:19:47.367093+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:19:47.367093+0000","last_undegraded":"2023-09-23T08:19:47.367093+0000","last_fullsized":"2023-09-23T08:19:47.367093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"94'4531","reported_seq":8690,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:19:48.970501+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:19:48.970501+0000","last_peered":"2023-09-23T08:19:48.970501+0000","last_clean":"2023-09-23T08:19:48.970501+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:19:48.970501+0000","last_undegraded":"2023-09-23T08:19:48.970501+0000","last_fullsized":"2023-09-23T08:19:48.970501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19995562003,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1011370,"num_read_kb":3962308,"num_write":998352,"num_write_kb":3993402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12287635456,"data_stored":54479904768,"data_compressed":36581648,"data_compressed_allocated":1684156416,"data_compressed_original":42461794304,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35555,"ondisk_log_size":35555,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739182,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5178516,"kb_used_data":4000460,"kb_used_omap":572,"kb_used_meta":1177475,"kb_avail":89193324,"statfs":{"total":96636764160,"available":91333963776,"internally_reserved":0,"allocated":4096471040,"data_stored":18160337995,"data_compressed":11796034,"data_compressed_allocated":561381376,"data_compressed_original":14154424320,"omap_allocated":585792,"internal_metadata":1205735360},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":2.056,"15min":2.1360000000000001},"min":{"1min":0.17999999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":11.561,"5min":13.019,"15min":16.388999999999999},"last":7.4690000000000003},{"interface":"front","average":{"1min":1.9039999999999999,"5min":2.1459999999999999,"15min":2.4750000000000001},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":11.597,"5min":22.266999999999999,"15min":22.266999999999999},"last":7.5300000000000002}]},{"osd":1,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":2.8319999999999999,"5min":3.4209999999999998,"15min":3.1419999999999999},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":15.848000000000001,"5min":19.433,"15min":23.045999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":3.6520000000000001,"5min":3.0099999999999998,"15min":3.0209999999999999},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":28.257000000000001,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739179,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4746336,"kb_used_data":4000984,"kb_used_omap":580,"kb_used_meta":744763,"kb_avail":89625504,"statfs":{"total":96636764160,"available":91776516096,"internally_reserved":0,"allocated":4097007616,"data_stored":18160961432,"data_compressed":12384766,"data_compressed_allocated":561422336,"data_compressed_original":14154645504,"omap_allocated":594767,"internal_metadata":762637489},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,4,8,5,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":2.7799999999999998,"15min":2.7469999999999999},"min":{"1min":0.26200000000000001,"5min":0.19400000000000001,"15min":0.17000000000000001},"max":{"1min":10.904999999999999,"5min":11.981999999999999,"15min":19.885999999999999},"last":0.38},{"interface":"front","average":{"1min":2.4990000000000001,"5min":3.351,"15min":2.9390000000000001},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":6.1429999999999998,"5min":14.66,"15min":24.385000000000002},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Sep 23 08:19:05 2023","interfaces":[{"interface":"back","average":{"1min":2.4329999999999998,"5min":3.383,"15min":3.3410000000000002},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.15,"5min":25.84,"15min":25.84},"last":0.57799999999999996},{"interface":"front","average":{"1min":2.5990000000000002,"5min":2.9710000000000001,"15min":2.7890000000000001},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.20699999999999999},"max":{"1min":13.435,"5min":25.120999999999999,"15min":25.120999999999999},"last":0.41399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739179,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5297236,"kb_used_data":4000972,"kb_used_omap":582,"kb_used_meta":1295673,"kb_avail":89074604,"statfs":{"total":96636764160,"available":91212394496,"internally_reserved":0,"allocated":4096995328,"data_stored":18160937238,"data_compressed":12406779,"data_compressed_allocated":561422336,"data_compressed_original":14154858496,"omap_allocated":596311,"internal_metadata":1326769833},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,24,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":2.5,"15min":2.5640000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":6.2729999999999997,"5min":13.217000000000001,"15min":14.496},"last":9.1219999999999999},{"interface":"front","average":{"1min":2.794,"5min":3.1400000000000001,"15min":3.3319999999999999},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.189},"max":{"1min":11.513999999999999,"5min":14.779,"15min":25.012},"last":9.0809999999999995}]},{"osd":2,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":4.5629999999999997,"5min":3.5880000000000001,"15min":3.286},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":18.414000000000001,"5min":20.216999999999999,"15min":23.443999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":2.484,"5min":1.917,"15min":2.0169999999999999},"min":{"1min":0.249,"5min":0.193,"15min":0.187},"max":{"1min":18.353000000000002,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4095844352,"data_stored":18159976448,"data_compressed":12381827,"data_compressed_allocated":561385472,"data_compressed_original":14153969664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096008192,"data_stored":18159968256,"data_compressed":12404009,"data_compressed_allocated":561385472,"data_compressed_original":14153809920,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4095782912,"data_stored":18159960064,"data_compressed":11795812,"data_compressed_allocated":561385472,"data_compressed_original":14154014720,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:19:51.726 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:51.726 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:51.726 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:51.727 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:19:51.727 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:20:11.728 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:20:14.671 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:20:14.690 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:20:14.703 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2138,"stamp":"2023-09-23T08:20:12.982779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20010563164,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1095748,"num_read_kb":4045408,"num_write":1022109,"num_write_kb":4030601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75916,"ondisk_log_size":75916,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15421600,"kb_used_data":12039492,"kb_used_omap":1735,"kb_used_meta":3380344,"kb_avail":267693920,"statfs":{"total":289910292480,"available":274118574080,"internally_reserved":0,"allocated":12328439808,"data_stored":54482990329,"data_compressed":36586225,"data_compressed_allocated":1684217856,"data_compressed_original":42428321792,"omap_allocated":1776870,"internal_metadata":3461473050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,12,7,17,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7208960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1880,"num_read_kb":7492,"num_write":2057,"num_write_kb":8228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003872"},"pg_stats":[{"pgid":"2.7","version":"94'5387","reported_seq":9031,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.848198+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:20:07.848198+0000","last_peered":"2023-09-23T08:20:07.848198+0000","last_clean":"2023-09-23T08:20:07.848198+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:20:07.848198+0000","last_undegraded":"2023-09-23T08:20:07.848198+0000","last_fullsized":"2023-09-23T08:20:07.848198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7889","reported_seq":13610,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.848415+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:20:07.848415+0000","last_peered":"2023-09-23T08:20:07.848415+0000","last_clean":"2023-09-23T08:20:07.848415+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:20:07.848415+0000","last_undegraded":"2023-09-23T08:20:07.848415+0000","last_fullsized":"2023-09-23T08:20:07.848415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3837","reported_seq":6279,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.848133+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:20:07.848133+0000","last_peered":"2023-09-23T08:20:07.848133+0000","last_clean":"2023-09-23T08:20:07.848133+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:20:07.848133+0000","last_undegraded":"2023-09-23T08:20:07.848133+0000","last_fullsized":"2023-09-23T08:20:07.848133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"94'257490","reported_seq":515587,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:09.334177+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:20:09.334177+0000","last_peered":"2023-09-23T08:20:09.334177+0000","last_clean":"2023-09-23T08:20:09.334177+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:20:09.334177+0000","last_undegraded":"2023-09-23T08:20:09.334177+0000","last_fullsized":"2023-09-23T08:20:09.334177+0000","mapping_epoch":18,"log_start":"91'247460","ondisk_log_start":"91'247460","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":480,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 72s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5114490880,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":259250,"num_read_kb":1015352,"num_write":256108,"num_write_kb":1024432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2964,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.855187+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:20:07.855187+0000","last_peered":"2023-09-23T08:20:07.855187+0000","last_clean":"2023-09-23T08:20:07.855187+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:20:07.855187+0000","last_undegraded":"2023-09-23T08:20:07.855187+0000","last_fullsized":"2023-09-23T08:20:07.855187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'243501","reported_seq":488017,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:09.766826+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:20:09.766826+0000","last_peered":"2023-09-23T08:20:09.766826+0000","last_clean":"2023-09-23T08:20:09.766826+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:20:09.766826+0000","last_undegraded":"2023-09-23T08:20:09.766826+0000","last_fullsized":"2023-09-23T08:20:09.766826+0000","mapping_epoch":18,"log_start":"91'233491","ondisk_log_start":"91'233491","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4846727187,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":245636,"num_read_kb":962452,"num_write":242200,"num_write_kb":968794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":263,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:53.202923+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:53.202923+0000","last_peered":"2023-09-23T08:18:53.202923+0000","last_clean":"2023-09-23T08:18:53.202923+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:53.202923+0000","last_undegraded":"2023-09-23T08:18:53.202923+0000","last_fullsized":"2023-09-23T08:18:53.202923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'6545","reported_seq":10131,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.848529+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:20:07.848529+0000","last_peered":"2023-09-23T08:20:07.848529+0000","last_clean":"2023-09-23T08:20:07.848529+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:20:07.848529+0000","last_undegraded":"2023-09-23T08:20:07.848529+0000","last_fullsized":"2023-09-23T08:20:07.848529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"94'245122","reported_seq":491384,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:09.699589+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:20:09.699589+0000","last_peered":"2023-09-23T08:20:09.699589+0000","last_clean":"2023-09-23T08:20:09.699589+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:20:09.699589+0000","last_undegraded":"2023-09-23T08:20:09.699589+0000","last_fullsized":"2023-09-23T08:20:09.699589+0000","mapping_epoch":18,"log_start":"91'235077","ondisk_log_start":"91'235077","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":543,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 63s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4864413696,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":247329,"num_read_kb":969060,"num_write":243830,"num_write_kb":975320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3633,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:02.727984+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:20:02.727984+0000","last_peered":"2023-09-23T08:20:02.727984+0000","last_clean":"2023-09-23T08:20:02.727984+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:20:02.727984+0000","last_undegraded":"2023-09-23T08:20:02.727984+0000","last_fullsized":"2023-09-23T08:20:02.727984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"94'261630","reported_seq":523495,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:09.337488+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:20:09.337488+0000","last_peered":"2023-09-23T08:20:09.337488+0000","last_clean":"2023-09-23T08:20:09.337488+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:20:09.337488+0000","last_undegraded":"2023-09-23T08:20:09.337488+0000","last_fullsized":"2023-09-23T08:20:09.337488+0000","mapping_epoch":18,"log_start":"91'251555","ondisk_log_start":"91'251555","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":473,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5184331776,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":262997,"num_read_kb":1030780,"num_write":260273,"num_write_kb":1041092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'4585","reported_seq":7007,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.864673+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:20:07.864673+0000","last_peered":"2023-09-23T08:20:07.864673+0000","last_clean":"2023-09-23T08:20:07.864673+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:20:07.864673+0000","last_undegraded":"2023-09-23T08:20:07.864673+0000","last_fullsized":"2023-09-23T08:20:07.864673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"94'4549","reported_seq":8720,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:07.848229+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:20:07.848229+0000","last_peered":"2023-09-23T08:20:07.848229+0000","last_clean":"2023-09-23T08:20:07.848229+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:20:07.848229+0000","last_undegraded":"2023-09-23T08:20:07.848229+0000","last_fullsized":"2023-09-23T08:20:07.848229+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20009963539,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1015212,"num_read_kb":3977644,"num_write":1002411,"num_write_kb":4009638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12325634048,"data_stored":54480707584,"data_compressed":36581015,"data_compressed_allocated":1684144128,"data_compressed_original":42426179584,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35657,"ondisk_log_size":35657,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739186,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5240624,"kb_used_data":4012584,"kb_used_omap":572,"kb_used_meta":1227459,"kb_avail":89131216,"statfs":{"total":96636764160,"available":91270365184,"internally_reserved":0,"allocated":4108886016,"data_stored":18160591947,"data_compressed":11794840,"data_compressed_allocated":561381376,"data_compressed_original":14142803968,"omap_allocated":585792,"internal_metadata":1256918976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":2.056,"15min":2.1360000000000001},"min":{"1min":0.17999999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":11.561,"5min":13.019,"15min":16.388999999999999},"last":1.726},{"interface":"front","average":{"1min":1.9039999999999999,"5min":2.1459999999999999,"15min":2.4750000000000001},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":11.597,"5min":22.266999999999999,"15min":22.266999999999999},"last":1.7569999999999999}]},{"osd":1,"last update":"Sat Sep 23 08:19:27 2023","interfaces":[{"interface":"back","average":{"1min":2.8319999999999999,"5min":3.4209999999999998,"15min":3.1419999999999999},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":15.848000000000001,"5min":19.433,"15min":23.045999999999999},"last":1.796},{"interface":"front","average":{"1min":3.6520000000000001,"5min":3.0099999999999998,"15min":3.0209999999999999},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":28.257000000000001,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739183,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4790360,"kb_used_data":4013072,"kb_used_omap":580,"kb_used_meta":776699,"kb_avail":89581480,"statfs":{"total":96636764160,"available":91731435520,"internally_reserved":0,"allocated":4109385728,"data_stored":18161174424,"data_compressed":12384582,"data_compressed_allocated":561418240,"data_compressed_original":14143004672,"omap_allocated":594767,"internal_metadata":795339953},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,1,10,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":2.1469999999999998,"15min":2.605},"min":{"1min":0.182,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":3.5760000000000001,"5min":11.048,"15min":19.885999999999999},"last":2.177},{"interface":"front","average":{"1min":2.956,"5min":3.2909999999999999,"15min":2.9729999999999999},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":11.028,"5min":14.66,"15min":24.385000000000002},"last":8.3800000000000008}]},{"osd":2,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1549999999999998,"5min":3.0110000000000001,"15min":3.3799999999999999},"min":{"1min":0.254,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.8859999999999992,"5min":11.853999999999999,"15min":25.84},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":2.1640000000000001,"15min":2.778},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.6879999999999997,"5min":13.435,"15min":25.120999999999999},"last":0.20699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739183,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5390616,"kb_used_data":4013836,"kb_used_omap":582,"kb_used_meta":1376185,"kb_avail":88981224,"statfs":{"total":96636764160,"available":91116773376,"internally_reserved":0,"allocated":4110168064,"data_stored":18161223958,"data_compressed":12406803,"data_compressed_allocated":561418240,"data_compressed_original":14142513152,"omap_allocated":596311,"internal_metadata":1409214121},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,3,7,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":2.5,"15min":2.5640000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":6.2729999999999997,"5min":13.217000000000001,"15min":14.496},"last":0.29099999999999998},{"interface":"front","average":{"1min":2.794,"5min":3.1400000000000001,"15min":3.3319999999999999},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.189},"max":{"1min":11.513999999999999,"5min":14.779,"15min":25.012},"last":0.316}]},{"osd":2,"last update":"Sat Sep 23 08:19:19 2023","interfaces":[{"interface":"back","average":{"1min":4.5629999999999997,"5min":3.5880000000000001,"15min":3.286},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":18.414000000000001,"5min":20.216999999999999,"15min":23.443999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":2.484,"5min":1.917,"15min":2.0169999999999999},"min":{"1min":0.249,"5min":0.193,"15min":0.187},"max":{"1min":18.353000000000002,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4108484608,"data_stored":18160222208,"data_compressed":12381971,"data_compressed_allocated":561381376,"data_compressed_original":14142103552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4108763136,"data_stored":18160250880,"data_compressed":12403976,"data_compressed_allocated":561381376,"data_compressed_original":14141861888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4108386304,"data_stored":18160234496,"data_compressed":11795068,"data_compressed_allocated":561381376,"data_compressed_original":14142214144,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:20:14.705 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:14.705 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:14.706 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:14.706 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:14.706 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:20:34.707 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:20:35.361 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:20:35.377 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:20:35.397 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2149,"stamp":"2023-09-23T08:20:34.997824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20026467932,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1099763,"num_read_kb":4061440,"num_write":1026666,"num_write_kb":4048829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76061,"ondisk_log_size":76061,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15695832,"kb_used_data":12083264,"kb_used_omap":1735,"kb_used_meta":3610808,"kb_avail":267419688,"statfs":{"total":289910292480,"available":273837760512,"internally_reserved":0,"allocated":12373262336,"data_stored":54483182841,"data_compressed":36586637,"data_compressed_allocated":1684242432,"data_compressed_original":42385403904,"omap_allocated":1776870,"internal_metadata":3697468186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,6,6,13,9,7,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8908800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2228,"num_read_kb":8900,"num_write":2549,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008071"},"pg_stats":[{"pgid":"2.7","version":"94'5393","reported_seq":9049,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.328921+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:20:28.328921+0000","last_peered":"2023-09-23T08:20:28.328921+0000","last_clean":"2023-09-23T08:20:28.328921+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:20:28.328921+0000","last_undegraded":"2023-09-23T08:20:28.328921+0000","last_fullsized":"2023-09-23T08:20:28.328921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7897","reported_seq":13634,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.329407+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:20:28.329407+0000","last_peered":"2023-09-23T08:20:28.329407+0000","last_clean":"2023-09-23T08:20:28.329407+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:20:28.329407+0000","last_undegraded":"2023-09-23T08:20:28.329407+0000","last_fullsized":"2023-09-23T08:20:28.329407+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7897,"log_dups_size":0,"ondisk_log_size":7897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3841","reported_seq":6291,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.330323+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:20:28.330323+0000","last_peered":"2023-09-23T08:20:28.330323+0000","last_clean":"2023-09-23T08:20:28.330323+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:20:28.330323+0000","last_undegraded":"2023-09-23T08:20:28.330323+0000","last_fullsized":"2023-09-23T08:20:28.330323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"94'258790","reported_seq":518067,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:34.792556+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:20:34.792556+0000","last_peered":"2023-09-23T08:20:34.792556+0000","last_clean":"2023-09-23T08:20:34.792556+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:20:34.792556+0000","last_undegraded":"2023-09-23T08:20:34.792556+0000","last_fullsized":"2023-09-23T08:20:34.792556+0000","mapping_epoch":18,"log_start":"92'248779","ondisk_log_start":"92'248779","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":643,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 98s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5119127552,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":260417,"num_read_kb":1020016,"num_write":257408,"num_write_kb":1029632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2975,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:33.453012+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:20:33.453012+0000","last_peered":"2023-09-23T08:20:33.453012+0000","last_clean":"2023-09-23T08:20:33.453012+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:20:33.453012+0000","last_undegraded":"2023-09-23T08:20:33.453012+0000","last_fullsized":"2023-09-23T08:20:33.453012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'244508","reported_seq":489861,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:30.059967+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:20:30.059967+0000","last_peered":"2023-09-23T08:20:30.059967+0000","last_clean":"2023-09-23T08:20:30.059967+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:20:30.059967+0000","last_undegraded":"2023-09-23T08:20:30.059967+0000","last_fullsized":"2023-09-23T08:20:30.059967+0000","mapping_epoch":18,"log_start":"91'234404","ondisk_log_start":"91'234404","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10104,"log_dups_size":0,"ondisk_log_size":10104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4850262035,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":246472,"num_read_kb":965792,"num_write":243207,"num_write_kb":972822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":263,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:53.202923+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:53.202923+0000","last_peered":"2023-09-23T08:18:53.202923+0000","last_clean":"2023-09-23T08:18:53.202923+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:53.202923+0000","last_undegraded":"2023-09-23T08:18:53.202923+0000","last_fullsized":"2023-09-23T08:18:53.202923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'6551","reported_seq":10149,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.335721+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:20:28.335721+0000","last_peered":"2023-09-23T08:20:28.335721+0000","last_clean":"2023-09-23T08:20:28.335721+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:20:28.335721+0000","last_undegraded":"2023-09-23T08:20:28.335721+0000","last_fullsized":"2023-09-23T08:20:28.335721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6551,"log_dups_size":0,"ondisk_log_size":6551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"94'246084","reported_seq":493241,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:30.042987+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:20:30.042987+0000","last_peered":"2023-09-23T08:20:30.042987+0000","last_clean":"2023-09-23T08:20:30.042987+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:20:30.042987+0000","last_undegraded":"2023-09-23T08:20:30.042987+0000","last_fullsized":"2023-09-23T08:20:30.042987+0000","mapping_epoch":18,"log_start":"91'235982","ondisk_log_start":"91'235982","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":662,"log_size":10102,"log_dups_size":0,"ondisk_log_size":10102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 83s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4867694592,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":248211,"num_read_kb":972576,"num_write":244792,"num_write_kb":979168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3638,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.331357+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:20:28.331357+0000","last_peered":"2023-09-23T08:20:28.331357+0000","last_clean":"2023-09-23T08:20:28.331357+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:20:28.331357+0000","last_undegraded":"2023-09-23T08:20:28.331357+0000","last_fullsized":"2023-09-23T08:20:28.331357+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"94'262918","reported_seq":525923,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:34.819543+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:20:34.819543+0000","last_peered":"2023-09-23T08:20:34.819543+0000","last_clean":"2023-09-23T08:20:34.819543+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:20:34.819543+0000","last_undegraded":"2023-09-23T08:20:34.819543+0000","last_fullsized":"2023-09-23T08:20:34.819543+0000","mapping_epoch":18,"log_start":"92'252877","ondisk_log_start":"92'252877","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":618,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5188784128,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":264127,"num_read_kb":1035292,"num_write":261561,"num_write_kb":1046244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'4602","reported_seq":7034,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:33.446726+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:20:33.446726+0000","last_peered":"2023-09-23T08:20:33.446726+0000","last_clean":"2023-09-23T08:20:33.446726+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:20:33.446726+0000","last_undegraded":"2023-09-23T08:20:33.446726+0000","last_fullsized":"2023-09-23T08:20:33.446726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"94'4555","reported_seq":8738,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:28.329930+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:20:28.329930+0000","last_peered":"2023-09-23T08:20:28.329930+0000","last_clean":"2023-09-23T08:20:28.329930+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:20:28.329930+0000","last_undegraded":"2023-09-23T08:20:28.329930+0000","last_fullsized":"2023-09-23T08:20:28.329930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20025868307,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1019227,"num_read_kb":3993676,"num_write":1006968,"num_write_kb":4027866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12369801216,"data_stored":54480863232,"data_compressed":36581349,"data_compressed_allocated":1684168704,"data_compressed_original":42383876096,"omap_allocated":0,"internal_metadata":0},"log_size":40258,"ondisk_log_size":40258,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35704,"ondisk_log_size":35704,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739191,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5328676,"kb_used_data":4028444,"kb_used_omap":572,"kb_used_meta":1299651,"kb_avail":89043164,"statfs":{"total":96636764160,"available":91180199936,"internally_reserved":0,"allocated":4125126656,"data_stored":18160677963,"data_compressed":11794749,"data_compressed_allocated":561389568,"data_compressed_original":14127259648,"omap_allocated":585792,"internal_metadata":1330843584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.6859999999999999,"15min":2.109},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":7.4690000000000003,"5min":11.561,"15min":16.388999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.774,"15min":2.3889999999999998},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":7.5300000000000002,"5min":11.597,"15min":22.266999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.387,"15min":3.2170000000000001},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":16.27,"5min":19.433,"15min":23.045999999999999},"last":6.2009999999999996},{"interface":"front","average":{"1min":2.3809999999999998,"5min":2.6760000000000002,"15min":3.0169999999999999},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.044,"5min":28.257000000000001,"15min":28.257000000000001},"last":1.577}]}]},{"osd":0,"up_from":8,"seq":34359739188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4914284,"kb_used_data":4028708,"kb_used_omap":580,"kb_used_meta":884987,"kb_avail":89457556,"statfs":{"total":96636764160,"available":91604537344,"internally_reserved":0,"allocated":4125396992,"data_stored":18161231768,"data_compressed":12384808,"data_compressed_allocated":561426432,"data_compressed_original":14127665152,"omap_allocated":594767,"internal_metadata":906226865},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,0,6,2,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":2.1469999999999998,"15min":2.605},"min":{"1min":0.182,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":3.5760000000000001,"5min":11.048,"15min":19.885999999999999},"last":0.311},{"interface":"front","average":{"1min":2.956,"5min":3.2909999999999999,"15min":2.9729999999999999},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":11.028,"5min":14.66,"15min":24.385000000000002},"last":4.1440000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1549999999999998,"5min":3.0110000000000001,"15min":3.3799999999999999},"min":{"1min":0.254,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.8859999999999992,"5min":11.853999999999999,"15min":25.84},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":2.1640000000000001,"15min":2.778},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.6879999999999997,"5min":13.435,"15min":25.120999999999999},"last":0.34499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739187,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5452872,"kb_used_data":4026112,"kb_used_omap":582,"kb_used_meta":1426169,"kb_avail":88918968,"statfs":{"total":96636764160,"available":91053023232,"internally_reserved":0,"allocated":4122738688,"data_stored":18161273110,"data_compressed":12407080,"data_compressed_allocated":561426432,"data_compressed_original":14130479104,"omap_allocated":596311,"internal_metadata":1460397737},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,7,3,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":2.4390000000000001,"15min":2.5920000000000001},"min":{"1min":0.245,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":11.404999999999999,"5min":13.217000000000001,"15min":14.496},"last":10.028},{"interface":"front","average":{"1min":3.1659999999999999,"5min":3.2639999999999998,"15min":3.3079999999999998},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":14.449,"5min":14.779,"15min":16.693000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":4.851,"5min":3.5230000000000001,"15min":3.4140000000000001},"min":{"1min":0.27700000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":17.849,"5min":18.414000000000001,"15min":23.443999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.704,"5min":1.829,"15min":2.004},"min":{"1min":0.253,"5min":0.193,"15min":0.187},"max":{"1min":7.0890000000000004,"5min":18.353000000000002,"15min":20.565000000000001},"last":8.5129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4124336128,"data_stored":18160300032,"data_compressed":12382296,"data_compressed_allocated":561389568,"data_compressed_original":14126944256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4121300992,"data_stored":18160287744,"data_compressed":12404204,"data_compressed_allocated":561389568,"data_compressed_original":14129844224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4124164096,"data_stored":18160275456,"data_compressed":11794849,"data_compressed_allocated":561389568,"data_compressed_original":14127087616,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:20:35.398 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:35.398 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:35.399 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:35.399 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:35.399 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:20:55.400 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:20:56.087 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:20:56.093 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:20:56.117 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2160,"stamp":"2023-09-23T08:20:55.525605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20040574556,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1103622,"num_read_kb":4076868,"num_write":1030683,"num_write_kb":4064897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76014,"ondisk_log_size":76014,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15947372,"kb_used_data":12119316,"kb_used_omap":1735,"kb_used_meta":3826296,"kb_avail":267168148,"statfs":{"total":289910292480,"available":273580183552,"internally_reserved":0,"allocated":12410179584,"data_stored":54483412217,"data_compressed":36587804,"data_compressed_allocated":1684291584,"data_compressed_original":42350080000,"omap_allocated":1776870,"internal_metadata":3918127898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,7,3,4,10,12,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7061504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1888,"num_read_kb":7544,"num_write":2007,"num_write_kb":8028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.511302"},"pg_stats":[{"pgid":"2.7","version":"94'5411","reported_seq":9079,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:48.814183+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:20:48.814183+0000","last_peered":"2023-09-23T08:20:48.814183+0000","last_clean":"2023-09-23T08:20:48.814183+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:20:48.814183+0000","last_undegraded":"2023-09-23T08:20:48.814183+0000","last_fullsized":"2023-09-23T08:20:48.814183+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7921","reported_seq":13674,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:48.810588+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:20:48.810588+0000","last_peered":"2023-09-23T08:20:48.810588+0000","last_clean":"2023-09-23T08:20:48.810588+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:20:48.810588+0000","last_undegraded":"2023-09-23T08:20:48.810588+0000","last_fullsized":"2023-09-23T08:20:48.810588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7921,"log_dups_size":0,"ondisk_log_size":7921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3853","reported_seq":6311,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:48.813651+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:20:48.813651+0000","last_peered":"2023-09-23T08:20:48.813651+0000","last_clean":"2023-09-23T08:20:48.813651+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:20:48.813651+0000","last_undegraded":"2023-09-23T08:20:48.813651+0000","last_fullsized":"2023-09-23T08:20:48.813651+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"94'259808","reported_seq":520050,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:55.198290+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:20:55.198290+0000","last_peered":"2023-09-23T08:20:55.198290+0000","last_clean":"2023-09-23T08:20:55.198290+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:20:55.198290+0000","last_undegraded":"2023-09-23T08:20:55.198290+0000","last_fullsized":"2023-09-23T08:20:55.198290+0000","mapping_epoch":18,"log_start":"92'249798","ondisk_log_start":"92'249798","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":745,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5122789376,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":261370,"num_read_kb":1023828,"num_write":258426,"num_write_kb":1033704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2983,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:53.969866+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:20:53.969866+0000","last_peered":"2023-09-23T08:20:53.969866+0000","last_clean":"2023-09-23T08:20:53.969866+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:20:53.969866+0000","last_undegraded":"2023-09-23T08:20:53.969866+0000","last_fullsized":"2023-09-23T08:20:53.969866+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'245535","reported_seq":491820,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:50.668652+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:20:50.668652+0000","last_peered":"2023-09-23T08:20:50.668652+0000","last_clean":"2023-09-23T08:20:50.668652+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:20:50.668652+0000","last_undegraded":"2023-09-23T08:20:50.668652+0000","last_fullsized":"2023-09-23T08:20:50.668652+0000","mapping_epoch":18,"log_start":"92'235518","ondisk_log_start":"92'235518","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4853915667,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":247405,"num_read_kb":969524,"num_write":244234,"num_write_kb":976930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":263,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-09-23T08:18:53.202923+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:18:53.202923+0000","last_peered":"2023-09-23T08:18:53.202923+0000","last_clean":"2023-09-23T08:18:53.202923+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:18:53.202923+0000","last_undegraded":"2023-09-23T08:18:53.202923+0000","last_fullsized":"2023-09-23T08:18:53.202923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'6569","reported_seq":10179,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:48.850808+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:20:48.850808+0000","last_peered":"2023-09-23T08:20:48.850808+0000","last_clean":"2023-09-23T08:20:48.850808+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:20:48.850808+0000","last_undegraded":"2023-09-23T08:20:48.850808+0000","last_fullsized":"2023-09-23T08:20:48.850808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"94'247026","reported_seq":495178,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:50.655470+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:20:50.655470+0000","last_peered":"2023-09-23T08:20:50.655470+0000","last_clean":"2023-09-23T08:20:50.655470+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:20:50.655470+0000","last_undegraded":"2023-09-23T08:20:50.655470+0000","last_fullsized":"2023-09-23T08:20:50.655470+0000","mapping_epoch":18,"log_start":"92'237000","ondisk_log_start":"92'237000","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":805,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 104s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4870877184,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":249196,"num_read_kb":976512,"num_write":245734,"num_write_kb":982936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3643,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:49.797345+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:20:49.797345+0000","last_peered":"2023-09-23T08:20:49.797345+0000","last_clean":"2023-09-23T08:20:49.797345+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:20:49.797345+0000","last_undegraded":"2023-09-23T08:20:49.797345+0000","last_fullsized":"2023-09-23T08:20:49.797345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"94'263948","reported_seq":527962,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:20:55.291399+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:20:55.291399+0000","last_peered":"2023-09-23T08:20:55.291399+0000","last_clean":"2023-09-23T08:20:55.291399+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:20:55.291399+0000","last_undegraded":"2023-09-23T08:20:55.291399+0000","last_fullsized":"2023-09-23T08:20:55.291399+0000","mapping_epoch":18,"log_start":"92'253898","ondisk_log_start":"92'253898","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":784,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5192392704,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":265115,"num_read_kb":1039240,"num_write":262591,"num_write_kb":1050364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'4620","reported_seq":7060,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:53.930292+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:20:53.930292+0000","last_peered":"2023-09-23T08:20:53.930292+0000","last_clean":"2023-09-23T08:20:53.930292+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:20:53.930292+0000","last_undegraded":"2023-09-23T08:20:53.930292+0000","last_fullsized":"2023-09-23T08:20:53.930292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"94'4573","reported_seq":8768,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-09-23T08:20:48.813254+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:20:48.813254+0000","last_peered":"2023-09-23T08:20:48.813254+0000","last_clean":"2023-09-23T08:20:48.813254+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:20:48.813254+0000","last_undegraded":"2023-09-23T08:20:48.813254+0000","last_fullsized":"2023-09-23T08:20:48.813254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20039974931,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1023086,"num_read_kb":4009104,"num_write":1010985,"num_write_kb":4043934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12407619584,"data_stored":54481092608,"data_compressed":36582491,"data_compressed_allocated":1684226048,"data_compressed_original":42347692032,"omap_allocated":0,"internal_metadata":0},"log_size":40103,"ondisk_log_size":40103,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35812,"ondisk_log_size":35812,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5391308,"kb_used_data":4040516,"kb_used_omap":572,"kb_used_meta":1350211,"kb_avail":88980532,"statfs":{"total":96636764160,"available":91116064768,"internally_reserved":0,"allocated":4137488384,"data_stored":18160739403,"data_compressed":11794997,"data_compressed_allocated":561405952,"data_compressed_original":14115414016,"omap_allocated":585792,"internal_metadata":1382617024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.6859999999999999,"15min":2.109},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":7.4690000000000003,"5min":11.561,"15min":16.388999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.774,"15min":2.3889999999999998},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":7.5300000000000002,"5min":11.597,"15min":22.266999999999999},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.387,"15min":3.2170000000000001},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":16.27,"5min":19.433,"15min":23.045999999999999},"last":4.4080000000000004},{"interface":"front","average":{"1min":2.3809999999999998,"5min":2.6760000000000002,"15min":3.0169999999999999},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.044,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.313}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4994176,"kb_used_data":4040632,"kb_used_omap":580,"kb_used_meta":952955,"kb_avail":89377664,"statfs":{"total":96636764160,"available":91522727936,"internally_reserved":0,"allocated":4137607168,"data_stored":18161350552,"data_compressed":12385494,"data_compressed_allocated":561442816,"data_compressed_original":14116016128,"omap_allocated":594767,"internal_metadata":975826097},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":2.1469999999999998,"15min":2.605},"min":{"1min":0.182,"5min":0.182,"15min":0.17000000000000001},"max":{"1min":3.5760000000000001,"5min":11.048,"15min":19.885999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.956,"5min":3.2909999999999999,"15min":2.9729999999999999},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":11.028,"5min":14.66,"15min":24.385000000000002},"last":1.071}]},{"osd":2,"last update":"Sat Sep 23 08:20:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1549999999999998,"5min":3.0110000000000001,"15min":3.3799999999999999},"min":{"1min":0.254,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.8859999999999992,"5min":11.853999999999999,"15min":25.84},"last":7.8209999999999997},{"interface":"front","average":{"1min":1.3300000000000001,"5min":2.1640000000000001,"15min":2.778},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.6879999999999997,"5min":13.435,"15min":25.120999999999999},"last":1.7669999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739191,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5561888,"kb_used_data":4038168,"kb_used_omap":582,"kb_used_meta":1523129,"kb_avail":88809952,"statfs":{"total":96636764160,"available":90941390848,"internally_reserved":0,"allocated":4135084032,"data_stored":18161322262,"data_compressed":12407313,"data_compressed_allocated":561442816,"data_compressed_original":14118649856,"omap_allocated":596311,"internal_metadata":1559684777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,0,1,4,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":2.4390000000000001,"15min":2.5920000000000001},"min":{"1min":0.245,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":11.404999999999999,"5min":13.217000000000001,"15min":14.496},"last":0.17399999999999999},{"interface":"front","average":{"1min":3.1659999999999999,"5min":3.2639999999999998,"15min":3.3079999999999998},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":14.449,"5min":14.779,"15min":16.693000000000001},"last":0.24299999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":4.851,"5min":3.5230000000000001,"15min":3.4140000000000001},"min":{"1min":0.27700000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":17.849,"5min":18.414000000000001,"15min":23.443999999999999},"last":2.7839999999999998},{"interface":"front","average":{"1min":1.704,"5min":1.829,"15min":2.004},"min":{"1min":0.253,"5min":0.193,"15min":0.187},"max":{"1min":7.0890000000000004,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4136902656,"data_stored":18160361472,"data_compressed":12382522,"data_compressed_allocated":561410048,"data_compressed_original":14114897920,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4134105088,"data_stored":18160386048,"data_compressed":12405100,"data_compressed_allocated":561405952,"data_compressed_original":14117625856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4136611840,"data_stored":18160345088,"data_compressed":11794869,"data_compressed_allocated":561410048,"data_compressed_original":14115168256,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:20:56.119 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:56.120 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:56.120 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:56.120 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:20:56.120 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:21:15.804 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:21:15.801+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:21:15.804 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:21:15.801+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:21:16.121 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:21:16.835 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:21:16.878 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:21:16.897 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2170,"stamp":"2023-09-23T08:21:15.040131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20052809308,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1106774,"num_read_kb":4089464,"num_write":1034205,"num_write_kb":4078985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76132,"ondisk_log_size":76132,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16019348,"kb_used_data":12151548,"kb_used_omap":1735,"kb_used_meta":3866040,"kb_avail":267096172,"statfs":{"total":289910292480,"available":273506480128,"internally_reserved":0,"allocated":12443185152,"data_stored":54483806294,"data_compressed":36590468,"data_compressed_allocated":1684422656,"data_compressed_original":42318880768,"omap_allocated":1776870,"internal_metadata":3958825754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,12,11,23,25,23,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7032832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1795,"num_read_kb":7168,"num_write":2018,"num_write_kb":8072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008511"},"pg_stats":[{"pgid":"2.7","version":"95'5417","reported_seq":9098,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.782503+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:21:10.782503+0000","last_peered":"2023-09-23T08:21:10.782503+0000","last_clean":"2023-09-23T08:21:10.782503+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:21:10.782503+0000","last_undegraded":"2023-09-23T08:21:10.782503+0000","last_fullsized":"2023-09-23T08:21:10.782503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7930","reported_seq":13700,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.801554+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:21:10.801554+0000","last_peered":"2023-09-23T08:21:10.801554+0000","last_clean":"2023-09-23T08:21:10.801554+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:21:10.801554+0000","last_undegraded":"2023-09-23T08:21:10.801554+0000","last_fullsized":"2023-09-23T08:21:10.801554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3857","reported_seq":6324,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.779453+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:21:10.779453+0000","last_peered":"2023-09-23T08:21:10.779453+0000","last_clean":"2023-09-23T08:21:10.779453+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:21:10.779453+0000","last_undegraded":"2023-09-23T08:21:10.779453+0000","last_fullsized":"2023-09-23T08:21:10.779453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"95'260594","reported_seq":521544,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:10.708569+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:21:10.708569+0000","last_peered":"2023-09-23T08:21:10.708569+0000","last_clean":"2023-09-23T08:21:10.708569+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:21:10.708569+0000","last_undegraded":"2023-09-23T08:21:10.708569+0000","last_fullsized":"2023-09-23T08:21:10.708569+0000","mapping_epoch":18,"log_start":"92'250508","ondisk_log_start":"92'250508","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":805,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 134s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5125574656,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":262068,"num_read_kb":1026620,"num_write":259212,"num_write_kb":1036848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":2990,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:09.297528+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:21:09.297528+0000","last_peered":"2023-09-23T08:21:09.297528+0000","last_clean":"2023-09-23T08:21:09.297528+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:21:09.297528+0000","last_undegraded":"2023-09-23T08:21:09.297528+0000","last_fullsized":"2023-09-23T08:21:09.297528+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'246497","reported_seq":493635,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.974397+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:21:10.974397+0000","last_peered":"2023-09-23T08:21:10.974397+0000","last_clean":"2023-09-23T08:21:10.974397+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:21:10.974397+0000","last_undegraded":"2023-09-23T08:21:10.974397+0000","last_fullsized":"2023-09-23T08:21:10.974397+0000","mapping_epoch":18,"log_start":"92'236434","ondisk_log_start":"92'236434","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4857286675,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":248258,"num_read_kb":972936,"num_write":245196,"num_write_kb":980778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":265,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:20:55.585622+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:20:55.585622+0000","last_peered":"2023-09-23T08:20:55.585622+0000","last_clean":"2023-09-23T08:20:55.585622+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:20:55.585622+0000","last_undegraded":"2023-09-23T08:20:55.585622+0000","last_fullsized":"2023-09-23T08:20:55.585622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'6575","reported_seq":10198,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.887787+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:21:10.887787+0000","last_peered":"2023-09-23T08:21:10.887787+0000","last_clean":"2023-09-23T08:21:10.887787+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:21:10.887787+0000","last_undegraded":"2023-09-23T08:21:10.887787+0000","last_fullsized":"2023-09-23T08:21:10.887787+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6575,"log_dups_size":0,"ondisk_log_size":6575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'248021","reported_seq":497062,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:10.977975+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:21:10.977975+0000","last_peered":"2023-09-23T08:21:10.977975+0000","last_clean":"2023-09-23T08:21:10.977975+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:21:10.977975+0000","last_undegraded":"2023-09-23T08:21:10.977975+0000","last_fullsized":"2023-09-23T08:21:10.977975+0000","mapping_epoch":18,"log_start":"92'238013","ondisk_log_start":"92'238013","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":931,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 124s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4874252288,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":250067,"num_read_kb":979988,"num_write":246729,"num_write_kb":986916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3647,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:04.167218+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:21:04.167218+0000","last_peered":"2023-09-23T08:21:04.167218+0000","last_clean":"2023-09-23T08:21:04.167218+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:21:04.167218+0000","last_undegraded":"2023-09-23T08:21:04.167218+0000","last_fullsized":"2023-09-23T08:21:04.167218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"95'264727","reported_seq":529482,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:10.733132+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:21:10.733132+0000","last_peered":"2023-09-23T08:21:10.733132+0000","last_clean":"2023-09-23T08:21:10.733132+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:21:10.733132+0000","last_undegraded":"2023-09-23T08:21:10.733132+0000","last_fullsized":"2023-09-23T08:21:10.733132+0000","mapping_epoch":18,"log_start":"92'254710","ondisk_log_start":"92'254710","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":858,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5195096064,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":265845,"num_read_kb":1042156,"num_write":263370,"num_write_kb":1053480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'4636","reported_seq":7083,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:09.708253+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:21:09.708253+0000","last_peered":"2023-09-23T08:21:09.708253+0000","last_clean":"2023-09-23T08:21:09.708253+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:21:09.708253+0000","last_undegraded":"2023-09-23T08:21:09.708253+0000","last_fullsized":"2023-09-23T08:21:09.708253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"95'4579","reported_seq":8787,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:10.801036+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:21:10.801036+0000","last_peered":"2023-09-23T08:21:10.801036+0000","last_clean":"2023-09-23T08:21:10.801036+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:21:10.801036+0000","last_undegraded":"2023-09-23T08:21:10.801036+0000","last_fullsized":"2023-09-23T08:21:10.801036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20052209683,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1026238,"num_read_kb":4021700,"num_write":1014507,"num_write_kb":4058022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12439859200,"data_stored":54481481728,"data_compressed":36584562,"data_compressed_allocated":1684348928,"data_compressed_original":42317213696,"omap_allocated":0,"internal_metadata":0},"log_size":40174,"ondisk_log_size":40174,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35859,"ondisk_log_size":35859,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739198,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5463580,"kb_used_data":4049876,"kb_used_omap":572,"kb_used_meta":1413123,"kb_avail":88908260,"statfs":{"total":96636764160,"available":91042058240,"internally_reserved":0,"allocated":4147073024,"data_stored":18160878954,"data_compressed":11795833,"data_compressed_allocated":561446912,"data_compressed_original":14106361856,"omap_allocated":585792,"internal_metadata":1447038912},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.6859999999999999,"15min":2.109},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":7.4690000000000003,"5min":11.561,"15min":16.388999999999999},"last":0.317},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.774,"15min":2.3889999999999998},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":7.5300000000000002,"5min":11.597,"15min":22.266999999999999},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Sep 23 08:20:29 2023","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.387,"15min":3.2170000000000001},"min":{"1min":0.216,"5min":0.189,"15min":0.186},"max":{"1min":16.27,"5min":19.433,"15min":23.045999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":2.3809999999999998,"5min":2.6760000000000002,"15min":3.0169999999999999},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.044,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739195,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4958876,"kb_used_data":4050580,"kb_used_omap":580,"kb_used_meta":907707,"kb_avail":89412964,"statfs":{"total":96636764160,"available":91558875136,"internally_reserved":0,"allocated":4147793920,"data_stored":18161453239,"data_compressed":12386208,"data_compressed_allocated":561487872,"data_compressed_original":14106390528,"omap_allocated":594767,"internal_metadata":929492145},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,4,13,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.8109999999999999,"15min":2.4289999999999998},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":4.3630000000000004,"5min":10.904999999999999,"15min":19.885999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.851,"5min":3.0489999999999999,"15min":2.9689999999999999},"min":{"1min":0.25700000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":10.176,"5min":14.66,"15min":24.385000000000002},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":3.496,"5min":3.0310000000000001,"15min":3.3809999999999998},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.637,"5min":11.637,"15min":25.84},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.8680000000000001,"15min":2.6619999999999999},"min":{"1min":0.26300000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.8090000000000002,"5min":13.435,"15min":25.120999999999999},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739195,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5596892,"kb_used_data":4051092,"kb_used_omap":582,"kb_used_meta":1545209,"kb_avail":88774948,"statfs":{"total":96636764160,"available":90905546752,"internally_reserved":0,"allocated":4148318208,"data_stored":18161474101,"data_compressed":12408427,"data_compressed_allocated":561487872,"data_compressed_original":14106128384,"omap_allocated":596311,"internal_metadata":1582294697},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,5,9,18,9,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":2.4390000000000001,"15min":2.5920000000000001},"min":{"1min":0.245,"5min":0.17100000000000001,"15min":0.14699999999999999},"max":{"1min":11.404999999999999,"5min":13.217000000000001,"15min":14.496},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.1659999999999999,"5min":3.2639999999999998,"15min":3.3079999999999998},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":14.449,"5min":14.779,"15min":16.693000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:20:20 2023","interfaces":[{"interface":"back","average":{"1min":4.851,"5min":3.5230000000000001,"15min":3.4140000000000001},"min":{"1min":0.27700000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":17.849,"5min":18.414000000000001,"15min":23.443999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.704,"5min":1.829,"15min":2.004},"min":{"1min":0.253,"5min":0.193,"15min":0.187},"max":{"1min":7.0890000000000004,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4146724864,"data_stored":18160517120,"data_compressed":12383702,"data_compressed_allocated":561451008,"data_compressed_original":14105661440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4146884608,"data_stored":18160508928,"data_compressed":12405687,"data_compressed_allocated":561451008,"data_compressed_original":14105505792,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4146249728,"data_stored":18160455680,"data_compressed":11795173,"data_compressed_allocated":561446912,"data_compressed_original":14106046464,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:21:16.899 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:16.899 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:16.899 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:16.900 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:16.900 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-09-23T08:21:17.570 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-09-23T08:21:17.586 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-09-23T08:21:18.293 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-09-23T08:21:18.317 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-09-23T08:21:19.024 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-09-23T08:21:19.045 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-09-23T08:21:19.653 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-09-23T08:21:19.675 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:21:39.676 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:21:40.368 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:21:40.391 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:21:40.409 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2182,"stamp":"2023-09-23T08:21:39.050471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20070504028,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1111600,"num_read_kb":4108720,"num_write":1039210,"num_write_kb":4099005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76245,"ondisk_log_size":76245,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16291100,"kb_used_data":12198144,"kb_used_omap":1735,"kb_used_meta":4091192,"kb_avail":266824420,"statfs":{"total":289910292480,"available":273228206080,"internally_reserved":0,"allocated":12490899456,"data_stored":54484375638,"data_compressed":36590697,"data_compressed_allocated":1684545536,"data_compressed_original":42273779712,"omap_allocated":1776870,"internal_metadata":4189381402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,7,3,9,28,19,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7139328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2056,"num_read_kb":8196,"num_write":2011,"num_write_kb":8044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007636"},"pg_stats":[{"pgid":"2.7","version":"95'5432","reported_seq":9128,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.886371+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:21:34.886371+0000","last_peered":"2023-09-23T08:21:34.886371+0000","last_clean":"2023-09-23T08:21:34.886371+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:21:34.886371+0000","last_undegraded":"2023-09-23T08:21:34.886371+0000","last_fullsized":"2023-09-23T08:21:34.886371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7950","reported_seq":13740,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.886585+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:21:34.886585+0000","last_peered":"2023-09-23T08:21:34.886585+0000","last_clean":"2023-09-23T08:21:34.886585+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:21:34.886585+0000","last_undegraded":"2023-09-23T08:21:34.886585+0000","last_fullsized":"2023-09-23T08:21:34.886585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7950,"log_dups_size":0,"ondisk_log_size":7950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3867","reported_seq":6344,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.885981+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:21:34.885981+0000","last_peered":"2023-09-23T08:21:34.885981+0000","last_clean":"2023-09-23T08:21:34.885981+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:21:34.885981+0000","last_undegraded":"2023-09-23T08:21:34.885981+0000","last_fullsized":"2023-09-23T08:21:34.885981+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"95'261867","reported_seq":524054,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:36.361030+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:21:36.361030+0000","last_peered":"2023-09-23T08:21:36.361030+0000","last_clean":"2023-09-23T08:21:36.361030+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:21:36.361030+0000","last_undegraded":"2023-09-23T08:21:36.361030+0000","last_fullsized":"2023-09-23T08:21:36.361030+0000","mapping_epoch":18,"log_start":"92'251828","ondisk_log_start":"92'251828","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":954,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 159s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5130080256,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":263288,"num_read_kb":1031500,"num_write":260485,"num_write_kb":1041940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3000,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.892594+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:21:34.892594+0000","last_peered":"2023-09-23T08:21:34.892594+0000","last_clean":"2023-09-23T08:21:34.892594+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:21:34.892594+0000","last_undegraded":"2023-09-23T08:21:34.892594+0000","last_fullsized":"2023-09-23T08:21:34.892594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'247688","reported_seq":495969,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:36.491429+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:21:36.491429+0000","last_peered":"2023-09-23T08:21:36.491429+0000","last_clean":"2023-09-23T08:21:36.491429+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:21:36.491429+0000","last_undegraded":"2023-09-23T08:21:36.491429+0000","last_fullsized":"2023-09-23T08:21:36.491429+0000","mapping_epoch":18,"log_start":"92'237649","ondisk_log_start":"92'237649","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4861517843,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":249397,"num_read_kb":977476,"num_write":246387,"num_write_kb":985542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":265,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:20:55.585622+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:20:55.585622+0000","last_peered":"2023-09-23T08:20:55.585622+0000","last_clean":"2023-09-23T08:20:55.585622+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:20:55.585622+0000","last_undegraded":"2023-09-23T08:20:55.585622+0000","last_fullsized":"2023-09-23T08:20:55.585622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'6590","reported_seq":10229,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.888671+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:21:34.888671+0000","last_peered":"2023-09-23T08:21:34.888671+0000","last_clean":"2023-09-23T08:21:34.888671+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:21:34.888671+0000","last_undegraded":"2023-09-23T08:21:34.888671+0000","last_fullsized":"2023-09-23T08:21:34.888671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6590,"log_dups_size":0,"ondisk_log_size":6590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'249219","reported_seq":499475,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:36.426351+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:21:36.426351+0000","last_peered":"2023-09-23T08:21:36.426351+0000","last_clean":"2023-09-23T08:21:36.426351+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:21:36.426351+0000","last_undegraded":"2023-09-23T08:21:36.426351+0000","last_fullsized":"2023-09-23T08:21:36.426351+0000","mapping_epoch":18,"log_start":"92'239136","ondisk_log_start":"92'239136","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1044,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 149s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4878462976,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":251269,"num_read_kb":984772,"num_write":247927,"num_write_kb":991708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3652,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:29.797740+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:21:29.797740+0000","last_peered":"2023-09-23T08:21:29.797740+0000","last_clean":"2023-09-23T08:21:29.797740+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:21:29.797740+0000","last_undegraded":"2023-09-23T08:21:29.797740+0000","last_fullsized":"2023-09-23T08:21:29.797740+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"95'266070","reported_seq":532106,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:36.417483+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:21:36.417483+0000","last_peered":"2023-09-23T08:21:36.417483+0000","last_clean":"2023-09-23T08:21:36.417483+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:21:36.417483+0000","last_undegraded":"2023-09-23T08:21:36.417483+0000","last_fullsized":"2023-09-23T08:21:36.417483+0000","mapping_epoch":18,"log_start":"92'256043","ondisk_log_start":"92'256043","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":949,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 163s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5199843328,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":267110,"num_read_kb":1047208,"num_write":264713,"num_write_kb":1058852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'4660","reported_seq":7118,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:36.259662+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:21:36.259662+0000","last_peered":"2023-09-23T08:21:36.259662+0000","last_clean":"2023-09-23T08:21:36.259662+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:21:36.259662+0000","last_undegraded":"2023-09-23T08:21:36.259662+0000","last_fullsized":"2023-09-23T08:21:36.259662+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"95'4594","reported_seq":8817,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:34.886406+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:21:34.886406+0000","last_peered":"2023-09-23T08:21:34.886406+0000","last_clean":"2023-09-23T08:21:34.886406+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:21:34.886406+0000","last_undegraded":"2023-09-23T08:21:34.886406+0000","last_fullsized":"2023-09-23T08:21:34.886406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20069904403,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1031064,"num_read_kb":4040956,"num_write":1019512,"num_write_kb":4078042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12488376320,"data_stored":54482071552,"data_compressed":36584948,"data_compressed_allocated":1684475904,"data_compressed_original":42271363072,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":35958,"ondisk_log_size":35958,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739203,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5551260,"kb_used_data":4065364,"kb_used_omap":572,"kb_used_meta":1485315,"kb_avail":88820580,"statfs":{"total":96636764160,"available":90952273920,"internally_reserved":0,"allocated":4162932736,"data_stored":18161022314,"data_compressed":11795519,"data_compressed_allocated":561487872,"data_compressed_original":14091345920,"omap_allocated":585792,"internal_metadata":1520963520},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.5189999999999999,"15min":2.0569999999999999},"min":{"1min":0.317,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":6.8940000000000001,"5min":11.561,"15min":13.019},"last":4.6150000000000002},{"interface":"front","average":{"1min":1.621,"5min":1.573,"15min":2.3100000000000001},"min":{"1min":0.35099999999999998,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":6.9340000000000002,"5min":11.597,"15min":22.266999999999999},"last":4.6710000000000003}]},{"osd":1,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":3.1819999999999999,"15min":3.177},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":13.222,"5min":19.433,"15min":23.045999999999999},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.077,"5min":2.323,"15min":2.8849999999999998},"min":{"1min":0.25700000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.4269999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":5.3540000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739200,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024732,"kb_used_data":4066132,"kb_used_omap":580,"kb_used_meta":958011,"kb_avail":89347108,"statfs":{"total":96636764160,"available":91491438592,"internally_reserved":0,"allocated":4163719168,"data_stored":18161670327,"data_compressed":12386589,"data_compressed_allocated":561528832,"data_compressed_original":14091354112,"omap_allocated":594767,"internal_metadata":981003441},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,0,3,11,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.8109999999999999,"15min":2.4289999999999998},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":4.3630000000000004,"5min":10.904999999999999,"15min":19.885999999999999},"last":0.314},{"interface":"front","average":{"1min":2.851,"5min":3.0489999999999999,"15min":2.9689999999999999},"min":{"1min":0.25700000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":10.176,"5min":14.66,"15min":24.385000000000002},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":3.496,"5min":3.0310000000000001,"15min":3.3809999999999998},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.637,"5min":11.637,"15min":25.84},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.8680000000000001,"15min":2.6619999999999999},"min":{"1min":0.26300000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.8090000000000002,"5min":13.435,"15min":25.120999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5715108,"kb_used_data":4066648,"kb_used_omap":582,"kb_used_meta":1647865,"kb_avail":88656732,"statfs":{"total":96636764160,"available":90784493568,"internally_reserved":0,"allocated":4164247552,"data_stored":18161682997,"data_compressed":12408589,"data_compressed_allocated":561528832,"data_compressed_original":14091079680,"omap_allocated":596311,"internal_metadata":1687414441},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,3,15,16,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.657,"15min":2.661},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":10.028,"5min":13.217000000000001,"15min":14.496},"last":8.6969999999999992},{"interface":"front","average":{"1min":2.8239999999999998,"5min":3.226,"15min":3.4100000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":15.09,"5min":15.09,"15min":16.693000000000001},"last":9.2530000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":3.7869999999999999,"5min":3.718,"15min":3.4780000000000002},"min":{"1min":0.27500000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":15.599,"5min":18.414000000000001,"15min":23.443999999999999},"last":3.1779999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.919,"15min":1.9630000000000001},"min":{"1min":0.218,"5min":0.193,"15min":0.187},"max":{"1min":8.5129999999999999,"5min":18.353000000000002,"15min":20.565000000000001},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4162998272,"data_stored":18160713728,"data_compressed":12383701,"data_compressed_allocated":561491968,"data_compressed_original":14090276864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4163018752,"data_stored":18160680960,"data_compressed":12405435,"data_compressed_allocated":561491968,"data_compressed_original":14090227712,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4162359296,"data_stored":18160676864,"data_compressed":11795812,"data_compressed_allocated":561491968,"data_compressed_original":14090858496,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:21:40.410 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:40.411 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:40.411 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:40.411 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:21:40.411 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:22:00.412 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:22:01.731 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:22:01.765 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:22:01.782 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2193,"stamp":"2023-09-23T08:22:01.078031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20084717148,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1115454,"num_read_kb":4124052,"num_write":1043211,"num_write_kb":4115009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76365,"ondisk_log_size":76365,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16176736,"kb_used_data":12240136,"kb_used_omap":1735,"kb_used_meta":3934840,"kb_avail":266938784,"statfs":{"total":289910292480,"available":273345314816,"internally_reserved":0,"allocated":12533899264,"data_stored":54485387350,"data_compressed":36590599,"data_compressed_allocated":1684721664,"data_compressed_original":42233507840,"omap_allocated":1776870,"internal_metadata":4029276954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,14,10,9,6,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7208960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1956,"num_read_kb":7772,"num_write":2004,"num_write_kb":8016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.019173"},"pg_stats":[{"pgid":"2.7","version":"95'5447","reported_seq":9156,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.369232+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:21:55.369232+0000","last_peered":"2023-09-23T08:21:55.369232+0000","last_clean":"2023-09-23T08:21:55.369232+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:21:55.369232+0000","last_undegraded":"2023-09-23T08:21:55.369232+0000","last_fullsized":"2023-09-23T08:21:55.369232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5447,"log_dups_size":0,"ondisk_log_size":5447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7970","reported_seq":13776,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.375150+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:21:55.375150+0000","last_peered":"2023-09-23T08:21:55.375150+0000","last_clean":"2023-09-23T08:21:55.375150+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:21:55.375150+0000","last_undegraded":"2023-09-23T08:21:55.375150+0000","last_fullsized":"2023-09-23T08:21:55.375150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3877","reported_seq":6363,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.369435+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:21:55.369435+0000","last_peered":"2023-09-23T08:21:55.369435+0000","last_clean":"2023-09-23T08:21:55.369435+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:21:55.369435+0000","last_undegraded":"2023-09-23T08:21:55.369435+0000","last_fullsized":"2023-09-23T08:21:55.369435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3877,"log_dups_size":0,"ondisk_log_size":3877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"95'262937","reported_seq":526160,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:56.895147+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:21:56.895147+0000","last_peered":"2023-09-23T08:21:56.895147+0000","last_clean":"2023-09-23T08:21:56.895147+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:21:56.895147+0000","last_undegraded":"2023-09-23T08:21:56.895147+0000","last_fullsized":"2023-09-23T08:21:56.895147+0000","mapping_epoch":18,"log_start":"93'252850","ondisk_log_start":"93'252850","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1068,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 180s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5133901824,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":264313,"num_read_kb":1035572,"num_write":261555,"num_write_kb":1046220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3010,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:22:00.544594+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:22:00.544594+0000","last_peered":"2023-09-23T08:22:00.544594+0000","last_clean":"2023-09-23T08:22:00.544594+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:22:00.544594+0000","last_undegraded":"2023-09-23T08:22:00.544594+0000","last_fullsized":"2023-09-23T08:22:00.544594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'248600","reported_seq":497801,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:56.787975+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:21:56.787975+0000","last_peered":"2023-09-23T08:21:56.787975+0000","last_clean":"2023-09-23T08:21:56.787975+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:21:56.787975+0000","last_undegraded":"2023-09-23T08:21:56.787975+0000","last_fullsized":"2023-09-23T08:21:56.787975+0000","mapping_epoch":18,"log_start":"92'238556","ondisk_log_start":"92'238556","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4864815123,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":250317,"num_read_kb":981136,"num_write":247299,"num_write_kb":989190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":265,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:20:55.585622+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:20:55.585622+0000","last_peered":"2023-09-23T08:20:55.585622+0000","last_clean":"2023-09-23T08:20:55.585622+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:20:55.585622+0000","last_undegraded":"2023-09-23T08:20:55.585622+0000","last_fullsized":"2023-09-23T08:20:55.585622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'6605","reported_seq":10256,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.380244+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:21:55.380244+0000","last_peered":"2023-09-23T08:21:55.380244+0000","last_clean":"2023-09-23T08:21:55.380244+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:21:55.380244+0000","last_undegraded":"2023-09-23T08:21:55.380244+0000","last_fullsized":"2023-09-23T08:21:55.380244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'250179","reported_seq":501385,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:56.813051+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:21:56.813051+0000","last_peered":"2023-09-23T08:21:56.813051+0000","last_clean":"2023-09-23T08:21:56.813051+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:21:56.813051+0000","last_undegraded":"2023-09-23T08:21:56.813051+0000","last_fullsized":"2023-09-23T08:21:56.813051+0000","mapping_epoch":18,"log_start":"92'240150","ondisk_log_start":"92'240150","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1238,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 170s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4881772544,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":252205,"num_read_kb":988500,"num_write":248887,"num_write_kb":995548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3657,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.366955+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:21:55.366955+0000","last_peered":"2023-09-23T08:21:55.366955+0000","last_clean":"2023-09-23T08:21:55.366955+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:21:55.366955+0000","last_undegraded":"2023-09-23T08:21:55.366955+0000","last_fullsized":"2023-09-23T08:21:55.366955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"95'267129","reported_seq":534146,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:21:56.921414+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:21:56.921414+0000","last_peered":"2023-09-23T08:21:56.921414+0000","last_clean":"2023-09-23T08:21:56.921414+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:21:56.921414+0000","last_undegraded":"2023-09-23T08:21:56.921414+0000","last_fullsized":"2023-09-23T08:21:56.921414+0000","mapping_epoch":18,"log_start":"93'257064","ondisk_log_start":"93'257064","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1054,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 184s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5203628032,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":268083,"num_read_kb":1051080,"num_write":265772,"num_write_kb":1063088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'4668","reported_seq":7134,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.368301+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:21:55.368301+0000","last_peered":"2023-09-23T08:21:55.368301+0000","last_clean":"2023-09-23T08:21:55.368301+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:21:55.368301+0000","last_undegraded":"2023-09-23T08:21:55.368301+0000","last_fullsized":"2023-09-23T08:21:55.368301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"95'4609","reported_seq":8845,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:21:55.370637+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:21:55.370637+0000","last_peered":"2023-09-23T08:21:55.370637+0000","last_clean":"2023-09-23T08:21:55.370637+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:21:55.370637+0000","last_undegraded":"2023-09-23T08:21:55.370637+0000","last_fullsized":"2023-09-23T08:21:55.370637+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20084117523,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1034918,"num_read_kb":4056288,"num_write":1023513,"num_write_kb":4094046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12530597888,"data_stored":54483017728,"data_compressed":36584117,"data_compressed_allocated":1684647936,"data_compressed_original":42231762944,"omap_allocated":0,"internal_metadata":0},"log_size":40225,"ondisk_log_size":40225,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36041,"ondisk_log_size":36041,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5635608,"kb_used_data":4081680,"kb_used_omap":572,"kb_used_meta":1553347,"kb_avail":88736232,"statfs":{"total":96636764160,"available":90865901568,"internally_reserved":0,"allocated":4179640320,"data_stored":18161452394,"data_compressed":11796548,"data_compressed_allocated":561557504,"data_compressed_original":14075731968,"omap_allocated":585792,"internal_metadata":1590628288},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.5189999999999999,"15min":2.0569999999999999},"min":{"1min":0.317,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":6.8940000000000001,"5min":11.561,"15min":13.019},"last":1.462},{"interface":"front","average":{"1min":1.621,"5min":1.573,"15min":2.3100000000000001},"min":{"1min":0.35099999999999998,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":6.9340000000000002,"5min":11.597,"15min":22.266999999999999},"last":1.512}]},{"osd":1,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":3.1819999999999999,"15min":3.177},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":13.222,"5min":19.433,"15min":23.045999999999999},"last":3.891},{"interface":"front","average":{"1min":1.077,"5min":2.323,"15min":2.8849999999999998},"min":{"1min":0.25700000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.4269999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":4.0890000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739204,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4781576,"kb_used_data":4079296,"kb_used_omap":580,"kb_used_meta":701691,"kb_avail":89590264,"statfs":{"total":96636764160,"available":91740430336,"internally_reserved":0,"allocated":4177199104,"data_stored":18161957047,"data_compressed":12386133,"data_compressed_allocated":561582080,"data_compressed_original":14078701568,"omap_allocated":594767,"internal_metadata":718531761},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,5,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.8109999999999999,"15min":2.4289999999999998},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":4.3630000000000004,"5min":10.904999999999999,"15min":19.885999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":2.851,"5min":3.0489999999999999,"15min":2.9689999999999999},"min":{"1min":0.25700000000000001,"5min":0.22600000000000001,"15min":0.16800000000000001},"max":{"1min":10.176,"5min":14.66,"15min":24.385000000000002},"last":1.552}]},{"osd":2,"last update":"Sat Sep 23 08:21:07 2023","interfaces":[{"interface":"back","average":{"1min":3.496,"5min":3.0310000000000001,"15min":3.3809999999999998},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":11.637,"5min":11.637,"15min":25.84},"last":1.474},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.8680000000000001,"15min":2.6619999999999999},"min":{"1min":0.26300000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.8090000000000002,"5min":13.435,"15min":25.120999999999999},"last":0.628}]}]},{"osd":1,"up_from":8,"seq":34359739204,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5759552,"kb_used_data":4079160,"kb_used_omap":582,"kb_used_meta":1679801,"kb_avail":88612288,"statfs":{"total":96636764160,"available":90738982912,"internally_reserved":0,"allocated":4177059840,"data_stored":18161977909,"data_compressed":12407918,"data_compressed_allocated":561582080,"data_compressed_original":14079074304,"omap_allocated":596311,"internal_metadata":1720116905},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.657,"15min":2.661},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":10.028,"5min":13.217000000000001,"15min":14.496},"last":3.7280000000000002},{"interface":"front","average":{"1min":2.8239999999999998,"5min":3.226,"15min":3.4100000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":15.09,"5min":15.09,"15min":16.693000000000001},"last":2.0800000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":3.7869999999999999,"5min":3.718,"15min":3.4780000000000002},"min":{"1min":0.27500000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":15.599,"5min":18.414000000000001,"15min":23.443999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.919,"15min":1.9630000000000001},"min":{"1min":0.218,"5min":0.193,"15min":0.187},"max":{"1min":8.5129999999999999,"5min":18.353000000000002,"15min":20.565000000000001},"last":2.7559999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4176076800,"data_stored":18161004544,"data_compressed":12383185,"data_compressed_allocated":561545216,"data_compressed_original":14078013440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4176003072,"data_stored":18160967680,"data_compressed":12404654,"data_compressed_allocated":561545216,"data_compressed_original":14078050304,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4178518016,"data_stored":18161045504,"data_compressed":11796278,"data_compressed_allocated":561557504,"data_compressed_original":14075699200,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:22:01.784 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:01.784 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:01.785 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:01.785 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:01.785 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:22:21.786 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:22:22.528 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:22:22.537 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:22:22.556 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2023-09-23T08:22:21.096589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20099020380,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1119526,"num_read_kb":4140268,"num_write":1047263,"num_write_kb":4131217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76333,"ondisk_log_size":76333,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16483948,"kb_used_data":12278728,"kb_used_omap":1735,"kb_used_meta":4203448,"kb_avail":266631572,"statfs":{"total":289910292480,"available":273030729728,"internally_reserved":0,"allocated":12573417472,"data_stored":54485998386,"data_compressed":36590204,"data_compressed_allocated":1684815872,"data_compressed_original":42196275200,"omap_allocated":1776870,"internal_metadata":4304331546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,5,18,17,21,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7135232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2020,"num_read_kb":8052,"num_write":2014,"num_write_kb":8056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012174"},"pg_stats":[{"pgid":"2.7","version":"96'5462","reported_seq":9184,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.083272+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:22:16.083272+0000","last_peered":"2023-09-23T08:22:16.083272+0000","last_clean":"2023-09-23T08:22:16.083272+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:22:16.083272+0000","last_undegraded":"2023-09-23T08:22:16.083272+0000","last_fullsized":"2023-09-23T08:22:16.083272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7990","reported_seq":13814,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.102640+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:22:16.102640+0000","last_peered":"2023-09-23T08:22:16.102640+0000","last_clean":"2023-09-23T08:22:16.102640+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:22:16.102640+0000","last_undegraded":"2023-09-23T08:22:16.102640+0000","last_fullsized":"2023-09-23T08:22:16.102640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3887","reported_seq":6382,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.083059+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:22:16.083059+0000","last_peered":"2023-09-23T08:22:16.083059+0000","last_clean":"2023-09-23T08:22:16.083059+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:22:16.083059+0000","last_undegraded":"2023-09-23T08:22:16.083059+0000","last_fullsized":"2023-09-23T08:22:16.083059+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"96'263976","reported_seq":528266,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:17.338015+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:22:17.338015+0000","last_peered":"2023-09-23T08:22:17.338015+0000","last_clean":"2023-09-23T08:22:17.338015+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:22:17.338015+0000","last_undegraded":"2023-09-23T08:22:17.338015+0000","last_fullsized":"2023-09-23T08:22:17.338015+0000","mapping_epoch":18,"log_start":"93'253975","ondisk_log_start":"93'253975","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1173,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 200s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5137530880,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":265367,"num_read_kb":1039776,"num_write":262594,"num_write_kb":1050376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3019,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:20.974654+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:22:20.974654+0000","last_peered":"2023-09-23T08:22:20.974654+0000","last_clean":"2023-09-23T08:22:20.974654+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:22:20.974654+0000","last_undegraded":"2023-09-23T08:22:20.974654+0000","last_fullsized":"2023-09-23T08:22:20.974654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'249631","reported_seq":499812,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:17.434454+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:22:17.434454+0000","last_peered":"2023-09-23T08:22:17.434454+0000","last_clean":"2023-09-23T08:22:17.434454+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:22:17.434454+0000","last_undegraded":"2023-09-23T08:22:17.434454+0000","last_fullsized":"2023-09-23T08:22:17.434454+0000","mapping_epoch":18,"log_start":"93'239567","ondisk_log_start":"93'239567","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4868538387,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":251295,"num_read_kb":985040,"num_write":248330,"num_write_kb":993314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":266,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:20:55.752824+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:20:55.752824+0000","last_peered":"2023-09-23T08:20:55.752824+0000","last_clean":"2023-09-23T08:20:55.752824+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:20:55.752824+0000","last_undegraded":"2023-09-23T08:20:55.752824+0000","last_fullsized":"2023-09-23T08:20:55.752824+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'6620","reported_seq":10284,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.107477+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:22:16.107477+0000","last_peered":"2023-09-23T08:22:16.107477+0000","last_clean":"2023-09-23T08:22:16.107477+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:22:16.107477+0000","last_undegraded":"2023-09-23T08:22:16.107477+0000","last_fullsized":"2023-09-23T08:22:16.107477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6620,"log_dups_size":0,"ondisk_log_size":6620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"96'251177","reported_seq":503409,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:17.399091+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:22:17.399091+0000","last_peered":"2023-09-23T08:22:17.399091+0000","last_clean":"2023-09-23T08:22:17.399091+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:22:17.399091+0000","last_undegraded":"2023-09-23T08:22:17.399091+0000","last_fullsized":"2023-09-23T08:22:17.399091+0000","mapping_epoch":18,"log_start":"93'241170","ondisk_log_start":"93'241170","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1390,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 190s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4885315584,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":253218,"num_read_kb":992532,"num_write":249885,"num_write_kb":999540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3662,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:15.846554+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:22:15.846554+0000","last_peered":"2023-09-23T08:22:15.846554+0000","last_clean":"2023-09-23T08:22:15.846554+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:22:15.846554+0000","last_undegraded":"2023-09-23T08:22:15.846554+0000","last_fullsized":"2023-09-23T08:22:15.846554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"96'268113","reported_seq":536171,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:17.370694+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:22:17.370694+0000","last_peered":"2023-09-23T08:22:17.370694+0000","last_clean":"2023-09-23T08:22:17.370694+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:22:17.370694+0000","last_undegraded":"2023-09-23T08:22:17.370694+0000","last_fullsized":"2023-09-23T08:22:17.370694+0000","mapping_epoch":18,"log_start":"93'258077","ondisk_log_start":"93'258077","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1194,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5207035904,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":269110,"num_read_kb":1055156,"num_write":266756,"num_write_kb":1067024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'4678","reported_seq":7153,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.545408+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:22:16.545408+0000","last_peered":"2023-09-23T08:22:16.545408+0000","last_clean":"2023-09-23T08:22:16.545408+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:22:16.545408+0000","last_undegraded":"2023-09-23T08:22:16.545408+0000","last_fullsized":"2023-09-23T08:22:16.545408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"96'4624","reported_seq":8873,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:16.093876+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:22:16.093876+0000","last_peered":"2023-09-23T08:22:16.093876+0000","last_clean":"2023-09-23T08:22:16.093876+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:22:16.093876+0000","last_undegraded":"2023-09-23T08:22:16.093876+0000","last_fullsized":"2023-09-23T08:22:16.093876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20098420755,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1038990,"num_read_kb":4072504,"num_write":1027565,"num_write_kb":4110254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12570001408,"data_stored":54483644416,"data_compressed":36584404,"data_compressed_allocated":1684742144,"data_compressed_original":42194632704,"omap_allocated":0,"internal_metadata":0},"log_size":40108,"ondisk_log_size":40108,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36126,"ondisk_log_size":36126,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739212,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5744436,"kb_used_data":4094380,"kb_used_omap":572,"kb_used_meta":1649475,"kb_avail":88627404,"statfs":{"total":96636764160,"available":90754461696,"internally_reserved":0,"allocated":4192645120,"data_stored":18161608286,"data_compressed":11795879,"data_compressed_allocated":561577984,"data_compressed_original":14063431680,"omap_allocated":585792,"internal_metadata":1689063360},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.5189999999999999,"15min":2.0569999999999999},"min":{"1min":0.317,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":6.8940000000000001,"5min":11.561,"15min":13.019},"last":0.376},{"interface":"front","average":{"1min":1.621,"5min":1.573,"15min":2.3100000000000001},"min":{"1min":0.35099999999999998,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":6.9340000000000002,"5min":11.597,"15min":22.266999999999999},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Sep 23 08:21:31 2023","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":3.1819999999999999,"15min":3.177},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":13.222,"5min":19.433,"15min":23.045999999999999},"last":7.0439999999999996},{"interface":"front","average":{"1min":1.077,"5min":2.323,"15min":2.8849999999999998},"min":{"1min":0.25700000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.4269999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":6.9930000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739208,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4898316,"kb_used_data":4091964,"kb_used_omap":580,"kb_used_meta":805755,"kb_avail":89473524,"statfs":{"total":96636764160,"available":91620888576,"internally_reserved":0,"allocated":4190171136,"data_stored":18162194859,"data_compressed":12386430,"data_compressed_allocated":561618944,"data_compressed_original":14066515968,"omap_allocated":594767,"internal_metadata":825093297},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,7,11,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.7470000000000001,"15min":2.2919999999999998},"min":{"1min":0.193,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":19.885999999999999},"last":0.307},{"interface":"front","average":{"1min":3.1509999999999998,"5min":2.8639999999999999,"15min":2.9870000000000001},"min":{"1min":0.40500000000000003,"5min":0.22800000000000001,"15min":0.16800000000000001},"max":{"1min":15.093,"5min":15.093,"15min":24.385000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8570000000000002,"5min":3.2269999999999999,"15min":3.4510000000000001},"min":{"1min":0.27800000000000002,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":15.138,"5min":15.138,"15min":25.84},"last":0.47199999999999998},{"interface":"front","average":{"1min":4.3979999999999997,"5min":2.3490000000000002,"15min":2.6850000000000001},"min":{"1min":0.34599999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":12.609,"5min":13.435,"15min":25.120999999999999},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739208,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5841196,"kb_used_data":4092384,"kb_used_omap":582,"kb_used_meta":1748217,"kb_avail":88530644,"statfs":{"total":96636764160,"available":90655379456,"internally_reserved":0,"allocated":4190601216,"data_stored":18162195241,"data_compressed":12407895,"data_compressed_allocated":561618944,"data_compressed_original":14066327552,"omap_allocated":596311,"internal_metadata":1790174889},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,8,5,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.657,"15min":2.661},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":10.028,"5min":13.217000000000001,"15min":14.496},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.8239999999999998,"5min":3.226,"15min":3.4100000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":15.09,"5min":15.09,"15min":16.693000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:21:23 2023","interfaces":[{"interface":"back","average":{"1min":3.7869999999999999,"5min":3.718,"15min":3.4780000000000002},"min":{"1min":0.27500000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":15.599,"5min":18.414000000000001,"15min":23.443999999999999},"last":2.8140000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.919,"15min":1.9630000000000001},"min":{"1min":0.218,"5min":0.193,"15min":0.187},"max":{"1min":8.5129999999999999,"5min":18.353000000000002,"15min":20.565000000000001},"last":2.5510000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4189224960,"data_stored":18161213440,"data_compressed":12383487,"data_compressed_allocated":561582080,"data_compressed_original":14065623040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4189274112,"data_stored":18161213440,"data_compressed":12405038,"data_compressed_allocated":561582080,"data_compressed_original":14065577984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4191502336,"data_stored":18161217536,"data_compressed":11795879,"data_compressed_allocated":561577984,"data_compressed_original":14063431680,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:22:22.558 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:22.558 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:22.558 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:22.558 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:22.559 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:22:42.559 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:22:43.415 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:22:43.425 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:22:43.446 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2215,"stamp":"2023-09-23T08:22:43.105663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20115076700,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1124262,"num_read_kb":4159116,"num_write":1051797,"num_write_kb":4149353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76533,"ondisk_log_size":76533,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16667752,"kb_used_data":12319568,"kb_used_omap":1735,"kb_used_meta":4346424,"kb_avail":266447768,"statfs":{"total":289910292480,"available":272842514432,"internally_reserved":0,"allocated":12615237632,"data_stored":54486276914,"data_compressed":36588247,"data_compressed_allocated":1684832256,"data_compressed_original":42156441600,"omap_allocated":1776870,"internal_metadata":4450738970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,5,14,12,14,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8830976,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2645,"num_read_kb":10524,"num_write":2520,"num_write_kb":10080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005651"},"pg_stats":[{"pgid":"2.7","version":"96'5489","reported_seq":9226,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:41.774941+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:22:41.774941+0000","last_peered":"2023-09-23T08:22:41.774941+0000","last_clean":"2023-09-23T08:22:41.774941+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:22:41.774941+0000","last_undegraded":"2023-09-23T08:22:41.774941+0000","last_fullsized":"2023-09-23T08:22:41.774941+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8026","reported_seq":13870,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:41.775026+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:22:41.775026+0000","last_peered":"2023-09-23T08:22:41.775026+0000","last_clean":"2023-09-23T08:22:41.775026+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:22:41.775026+0000","last_undegraded":"2023-09-23T08:22:41.775026+0000","last_fullsized":"2023-09-23T08:22:41.775026+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8026,"log_dups_size":0,"ondisk_log_size":8026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3905","reported_seq":6410,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:41.769448+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:22:41.769448+0000","last_peered":"2023-09-23T08:22:41.769448+0000","last_clean":"2023-09-23T08:22:41.769448+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:22:41.769448+0000","last_undegraded":"2023-09-23T08:22:41.769448+0000","last_fullsized":"2023-09-23T08:22:41.769448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"96'265004","reported_seq":530402,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:38.235622+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:22:38.235622+0000","last_peered":"2023-09-23T08:22:38.235622+0000","last_clean":"2023-09-23T08:22:38.235622+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:22:38.235622+0000","last_undegraded":"2023-09-23T08:22:38.235622+0000","last_fullsized":"2023-09-23T08:22:38.235622+0000","mapping_epoch":18,"log_start":"93'254994","ondisk_log_start":"93'254994","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1293,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 221s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5141155840,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":266465,"num_read_kb":1044140,"num_write":263622,"num_write_kb":1054488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3026,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:36.337530+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:22:36.337530+0000","last_peered":"2023-09-23T08:22:36.337530+0000","last_clean":"2023-09-23T08:22:36.337530+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:22:36.337530+0000","last_undegraded":"2023-09-23T08:22:36.337530+0000","last_fullsized":"2023-09-23T08:22:36.337530+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'250853","reported_seq":502284,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:42.956162+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:22:42.956162+0000","last_peered":"2023-09-23T08:22:42.956162+0000","last_clean":"2023-09-23T08:22:42.956162+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:22:42.956162+0000","last_undegraded":"2023-09-23T08:22:42.956162+0000","last_fullsized":"2023-09-23T08:22:42.956162+0000","mapping_epoch":18,"log_start":"93'240786","ondisk_log_start":"93'240786","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4872912915,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":252548,"num_read_kb":990028,"num_write":249552,"num_write_kb":998202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":266,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-09-23T08:20:55.752824+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:20:55.752824+0000","last_peered":"2023-09-23T08:20:55.752824+0000","last_clean":"2023-09-23T08:20:55.752824+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:20:55.752824+0000","last_undegraded":"2023-09-23T08:20:55.752824+0000","last_fullsized":"2023-09-23T08:20:55.752824+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'6647","reported_seq":10326,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:41.796712+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:22:41.796712+0000","last_peered":"2023-09-23T08:22:41.796712+0000","last_clean":"2023-09-23T08:22:41.796712+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:22:41.796712+0000","last_undegraded":"2023-09-23T08:22:41.796712+0000","last_fullsized":"2023-09-23T08:22:41.796712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"96'252396","reported_seq":505896,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:42.958472+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:22:42.958472+0000","last_peered":"2023-09-23T08:22:42.958472+0000","last_clean":"2023-09-23T08:22:42.958472+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:22:42.958472+0000","last_undegraded":"2023-09-23T08:22:42.958472+0000","last_fullsized":"2023-09-23T08:22:42.958472+0000","mapping_epoch":18,"log_start":"93'242393","ondisk_log_start":"93'242393","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1588,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4889513984,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":254467,"num_read_kb":997508,"num_write":251104,"num_write_kb":1004416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3665,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:31.205873+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:22:31.205873+0000","last_peered":"2023-09-23T08:22:31.205873+0000","last_clean":"2023-09-23T08:22:31.205873+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:22:31.205873+0000","last_undegraded":"2023-09-23T08:22:31.205873+0000","last_fullsized":"2023-09-23T08:22:31.205873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"96'269178","reported_seq":538384,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:38.249965+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:22:38.249965+0000","last_peered":"2023-09-23T08:22:38.249965+0000","last_clean":"2023-09-23T08:22:38.249965+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:22:38.249965+0000","last_undegraded":"2023-09-23T08:22:38.249965+0000","last_fullsized":"2023-09-23T08:22:38.249965+0000","mapping_epoch":18,"log_start":"93'259095","ondisk_log_start":"93'259095","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1297,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 225s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5210894336,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":270246,"num_read_kb":1059676,"num_write":267821,"num_write_kb":1071284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'4688","reported_seq":7171,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:36.330079+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:22:36.330079+0000","last_peered":"2023-09-23T08:22:36.330079+0000","last_clean":"2023-09-23T08:22:36.330079+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:22:36.330079+0000","last_undegraded":"2023-09-23T08:22:36.330079+0000","last_fullsized":"2023-09-23T08:22:36.330079+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"96'4651","reported_seq":8915,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:41.775233+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:22:41.775233+0000","last_peered":"2023-09-23T08:22:41.775233+0000","last_clean":"2023-09-23T08:22:41.775233+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:22:41.775233+0000","last_undegraded":"2023-09-23T08:22:41.775233+0000","last_fullsized":"2023-09-23T08:22:41.775233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20114477075,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1043726,"num_read_kb":4091352,"num_write":1032099,"num_write_kb":4128390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12612972544,"data_stored":54483972096,"data_compressed":36582271,"data_compressed_allocated":1684766720,"data_compressed_original":42153791488,"omap_allocated":0,"internal_metadata":0},"log_size":40163,"ondisk_log_size":40163,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36271,"ondisk_log_size":36271,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5779144,"kb_used_data":4106880,"kb_used_omap":572,"kb_used_meta":1671683,"kb_avail":88592696,"statfs":{"total":96636764160,"available":90718920704,"internally_reserved":0,"allocated":4205445120,"data_stored":18161682014,"data_compressed":11794989,"data_compressed_allocated":561586176,"data_compressed_original":14051237888,"omap_allocated":585792,"internal_metadata":1711804352},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":1.702,"15min":2.0790000000000002},"min":{"1min":0.28699999999999998,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":9.9939999999999998,"5min":11.561,"15min":13.019},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.4119999999999999,"5min":1.756,"15min":2.2749999999999999},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.185},"max":{"1min":10.034000000000001,"5min":11.597,"15min":22.266999999999999},"last":1.054}]},{"osd":1,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":3.4260000000000002,"5min":3.0630000000000002,"15min":3.198},"min":{"1min":0.216,"5min":0.215,"15min":0.186},"max":{"1min":14.371,"5min":16.27,"15min":23.045999999999999},"last":8.8409999999999993},{"interface":"front","average":{"1min":2.8620000000000001,"5min":2.5369999999999999,"15min":2.8650000000000002},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.916,"5min":28.257000000000001,"15min":28.257000000000001},"last":5.5810000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739212,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4959656,"kb_used_data":4104736,"kb_used_omap":580,"kb_used_meta":854331,"kb_avail":89412184,"statfs":{"total":96636764160,"available":91558076416,"internally_reserved":0,"allocated":4203249664,"data_stored":18162293163,"data_compressed":12386087,"data_compressed_allocated":561623040,"data_compressed_original":14054064128,"omap_allocated":594767,"internal_metadata":874835121},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,7,8,11,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.7470000000000001,"15min":2.2919999999999998},"min":{"1min":0.193,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":19.885999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":3.1509999999999998,"5min":2.8639999999999999,"15min":2.9870000000000001},"min":{"1min":0.40500000000000003,"5min":0.22800000000000001,"15min":0.16800000000000001},"max":{"1min":15.093,"5min":15.093,"15min":24.385000000000002},"last":1.6579999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8570000000000002,"5min":3.2269999999999999,"15min":3.4510000000000001},"min":{"1min":0.27800000000000002,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":15.138,"5min":15.138,"15min":25.84},"last":2.4950000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":2.3490000000000002,"15min":2.6850000000000001},"min":{"1min":0.34599999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":12.609,"5min":13.435,"15min":25.120999999999999},"last":2.1970000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739213,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5928952,"kb_used_data":4107952,"kb_used_omap":582,"kb_used_meta":1820409,"kb_avail":88442888,"statfs":{"total":96636764160,"available":90565517312,"internally_reserved":0,"allocated":4206542848,"data_stored":18162301737,"data_compressed":12407171,"data_compressed_allocated":561623040,"data_compressed_original":14051139584,"omap_allocated":596311,"internal_metadata":1864099497},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.3159999999999998,"15min":2.6160000000000001},"min":{"1min":0.26800000000000002,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":8.6969999999999992,"5min":11.404999999999999,"15min":14.496},"last":9.8979999999999997},{"interface":"front","average":{"1min":2.6459999999999999,"5min":2.9100000000000001,"15min":3.306},"min":{"1min":0.24299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":12.968999999999999,"5min":15.09,"15min":16.693000000000001},"last":2.806}]},{"osd":2,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":3.327,"5min":3.843,"15min":3.367},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":11.367000000000001,"5min":18.414000000000001,"15min":20.216999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.365,"5min":1.72,"15min":1.7969999999999999},"min":{"1min":0.26300000000000001,"5min":0.218,"15min":0.187},"max":{"1min":7.4359999999999999,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.99399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4202639360,"data_stored":18161303552,"data_compressed":12383036,"data_compressed_allocated":561586176,"data_compressed_original":14052855808,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4205666304,"data_stored":18161356800,"data_compressed":12404365,"data_compressed_allocated":561594368,"data_compressed_original":14050017280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4204666880,"data_stored":18161311744,"data_compressed":11794870,"data_compressed_allocated":561586176,"data_compressed_original":14050918400,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:22:43.448 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:43.448 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:43.448 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:43.449 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:22:43.449 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:23:03.450 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:23:04.172 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:23:04.201 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:23:04.219 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2225,"stamp":"2023-09-23T08:23:03.119209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20127376988,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1127943,"num_read_kb":4173752,"num_write":1055242,"num_write_kb":4163133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76731,"ondisk_log_size":76731,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16953848,"kb_used_data":12354460,"kb_used_omap":1735,"kb_used_meta":4597624,"kb_avail":266161672,"statfs":{"total":289910292480,"available":272549552128,"internally_reserved":0,"allocated":12650967040,"data_stored":54487014194,"data_compressed":36589896,"data_compressed_allocated":1684967424,"data_compressed_original":42123120640,"omap_allocated":1776870,"internal_metadata":4707967770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,3,5,26,22,19,11,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6889472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2053,"num_read_kb":8168,"num_write":1939,"num_write_kb":7756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010696"},"pg_stats":[{"pgid":"2.7","version":"96'5507","reported_seq":9253,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.024647+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:22:58.024647+0000","last_peered":"2023-09-23T08:22:58.024647+0000","last_clean":"2023-09-23T08:22:58.024647+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:22:58.024647+0000","last_undegraded":"2023-09-23T08:22:58.024647+0000","last_fullsized":"2023-09-23T08:22:58.024647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8050","reported_seq":13906,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.032791+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:22:58.032791+0000","last_peered":"2023-09-23T08:22:58.032791+0000","last_clean":"2023-09-23T08:22:58.032791+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:22:58.032791+0000","last_undegraded":"2023-09-23T08:22:58.032791+0000","last_fullsized":"2023-09-23T08:22:58.032791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8050,"log_dups_size":0,"ondisk_log_size":8050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3917","reported_seq":6428,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.022595+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:22:58.022595+0000","last_peered":"2023-09-23T08:22:58.022595+0000","last_clean":"2023-09-23T08:22:58.022595+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:22:58.022595+0000","last_undegraded":"2023-09-23T08:22:58.022595+0000","last_fullsized":"2023-09-23T08:22:58.022595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"96'266007","reported_seq":532464,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:58.589077+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:22:58.589077+0000","last_peered":"2023-09-23T08:22:58.589077+0000","last_clean":"2023-09-23T08:22:58.589077+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:22:58.589077+0000","last_undegraded":"2023-09-23T08:22:58.589077+0000","last_fullsized":"2023-09-23T08:22:58.589077+0000","mapping_epoch":18,"log_start":"93'256004","ondisk_log_start":"93'256004","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1381,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 242s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5144735744,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":267510,"num_read_kb":1048288,"num_write":264625,"num_write_kb":1058500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3034,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:56.809861+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:22:56.809861+0000","last_peered":"2023-09-23T08:22:56.809861+0000","last_clean":"2023-09-23T08:22:56.809861+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:22:56.809861+0000","last_undegraded":"2023-09-23T08:22:56.809861+0000","last_fullsized":"2023-09-23T08:22:56.809861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'251572","reported_seq":503790,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.353097+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:22:58.353097+0000","last_peered":"2023-09-23T08:22:58.353097+0000","last_clean":"2023-09-23T08:22:58.353097+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:22:58.353097+0000","last_undegraded":"2023-09-23T08:22:58.353097+0000","last_fullsized":"2023-09-23T08:22:58.353097+0000","mapping_epoch":18,"log_start":"93'241501","ondisk_log_start":"93'241501","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4875558931,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":253333,"num_read_kb":993156,"num_write":250271,"num_write_kb":1001078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":267,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:05.613517+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:22:05.613517+0000","last_peered":"2023-09-23T08:22:05.613517+0000","last_clean":"2023-09-23T08:22:05.613517+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:22:05.613517+0000","last_undegraded":"2023-09-23T08:22:05.613517+0000","last_fullsized":"2023-09-23T08:22:05.613517+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'6665","reported_seq":10353,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.141723+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:22:58.141723+0000","last_peered":"2023-09-23T08:22:58.141723+0000","last_clean":"2023-09-23T08:22:58.141723+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:22:58.141723+0000","last_undegraded":"2023-09-23T08:22:58.141723+0000","last_fullsized":"2023-09-23T08:22:58.141723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6665,"log_dups_size":0,"ondisk_log_size":6665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"96'253097","reported_seq":507392,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:58.303739+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:22:58.303739+0000","last_peered":"2023-09-23T08:22:58.303739+0000","last_clean":"2023-09-23T08:22:58.303739+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:22:58.303739+0000","last_undegraded":"2023-09-23T08:22:58.303739+0000","last_fullsized":"2023-09-23T08:22:58.303739+0000","mapping_epoch":18,"log_start":"93'243004","ondisk_log_start":"93'243004","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1707,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 231s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4891987968,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":255258,"num_read_kb":1000648,"num_write":251805,"num_write_kb":1007220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3670,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:51.694572+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:22:51.694572+0000","last_peered":"2023-09-23T08:22:51.694572+0000","last_clean":"2023-09-23T08:22:51.694572+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:22:51.694572+0000","last_undegraded":"2023-09-23T08:22:51.694572+0000","last_fullsized":"2023-09-23T08:22:51.694572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"96'270200","reported_seq":540478,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:22:58.582361+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:22:58.582361+0000","last_peered":"2023-09-23T08:22:58.582361+0000","last_clean":"2023-09-23T08:22:58.582361+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:22:58.582361+0000","last_undegraded":"2023-09-23T08:22:58.582361+0000","last_fullsized":"2023-09-23T08:22:58.582361+0000","mapping_epoch":18,"log_start":"93'260108","ondisk_log_start":"93'260108","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1403,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 246s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5214494720,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":271306,"num_read_kb":1063896,"num_write":268843,"num_write_kb":1075372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'4700","reported_seq":7191,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:56.815368+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:22:56.815368+0000","last_peered":"2023-09-23T08:22:56.815368+0000","last_clean":"2023-09-23T08:22:56.815368+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:22:56.815368+0000","last_undegraded":"2023-09-23T08:22:56.815368+0000","last_fullsized":"2023-09-23T08:22:56.815368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"96'4669","reported_seq":8942,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.020920+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:22:58.020920+0000","last_peered":"2023-09-23T08:22:58.020920+0000","last_clean":"2023-09-23T08:22:58.020920+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:22:58.020920+0000","last_undegraded":"2023-09-23T08:22:58.020920+0000","last_fullsized":"2023-09-23T08:22:58.020920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20126777363,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1047407,"num_read_kb":4105988,"num_write":1035544,"num_write_kb":4142170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12647993344,"data_stored":54484705280,"data_compressed":36583769,"data_compressed_allocated":1684897792,"data_compressed_original":42121129984,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36373,"ondisk_log_size":36373,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739220,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5857876,"kb_used_data":4118988,"kb_used_omap":572,"kb_used_meta":1738307,"kb_avail":88513964,"statfs":{"total":96636764160,"available":90638299136,"internally_reserved":0,"allocated":4217843712,"data_stored":18161968734,"data_compressed":11795065,"data_compressed_allocated":561639424,"data_compressed_original":14039707648,"omap_allocated":585792,"internal_metadata":1780027328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":1.702,"15min":2.0790000000000002},"min":{"1min":0.28699999999999998,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":9.9939999999999998,"5min":11.561,"15min":13.019},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.4119999999999999,"5min":1.756,"15min":2.2749999999999999},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.185},"max":{"1min":10.034000000000001,"5min":11.597,"15min":22.266999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":3.4260000000000002,"5min":3.0630000000000002,"15min":3.198},"min":{"1min":0.216,"5min":0.215,"15min":0.186},"max":{"1min":14.371,"5min":16.27,"15min":23.045999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":2.8620000000000001,"5min":2.5369999999999999,"15min":2.8650000000000002},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.916,"5min":28.257000000000001,"15min":28.257000000000001},"last":1.095}]}]},{"osd":0,"up_from":8,"seq":34359739216,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076656,"kb_used_data":4117672,"kb_used_omap":580,"kb_used_meta":958395,"kb_avail":89295184,"statfs":{"total":96636764160,"available":91438268416,"internally_reserved":0,"allocated":4216496128,"data_stored":18162518443,"data_compressed":12386835,"data_compressed_allocated":561664000,"data_compressed_original":14041653248,"omap_allocated":594767,"internal_metadata":981396657},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,8,7,10,11,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.7470000000000001,"15min":2.2919999999999998},"min":{"1min":0.193,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":19.885999999999999},"last":4.6959999999999997},{"interface":"front","average":{"1min":3.1509999999999998,"5min":2.8639999999999999,"15min":2.9870000000000001},"min":{"1min":0.40500000000000003,"5min":0.22800000000000001,"15min":0.16800000000000001},"max":{"1min":15.093,"5min":15.093,"15min":24.385000000000002},"last":2.1000000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:22:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8570000000000002,"5min":3.2269999999999999,"15min":3.4510000000000001},"min":{"1min":0.27800000000000002,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":15.138,"5min":15.138,"15min":25.84},"last":4.7850000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":2.3490000000000002,"15min":2.6850000000000001},"min":{"1min":0.34599999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":12.609,"5min":13.435,"15min":25.120999999999999},"last":4.7569999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6019316,"kb_used_data":4117800,"kb_used_omap":582,"kb_used_meta":1900921,"kb_avail":88352524,"statfs":{"total":96636764160,"available":90472984576,"internally_reserved":0,"allocated":4216627200,"data_stored":18162527017,"data_compressed":12407996,"data_compressed_allocated":561664000,"data_compressed_original":14041759744,"omap_allocated":596311,"internal_metadata":1946543785},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,1,1,15,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.3159999999999998,"15min":2.6160000000000001},"min":{"1min":0.26800000000000002,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":8.6969999999999992,"5min":11.404999999999999,"15min":14.496},"last":0.84499999999999997},{"interface":"front","average":{"1min":2.6459999999999999,"5min":2.9100000000000001,"15min":3.306},"min":{"1min":0.24299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":12.968999999999999,"5min":15.09,"15min":16.693000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":3.327,"5min":3.843,"15min":3.367},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":11.367000000000001,"5min":18.414000000000001,"15min":20.216999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.72,"15min":1.7969999999999999},"min":{"1min":0.26300000000000001,"5min":0.218,"15min":0.187},"max":{"1min":7.4359999999999999,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.90300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4215402496,"data_stored":18161545216,"data_compressed":12383745,"data_compressed_allocated":561627136,"data_compressed_original":14040915968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4215242752,"data_stored":18161532928,"data_compressed":12405025,"data_compressed_allocated":561627136,"data_compressed_original":14041055232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4217348096,"data_stored":18161627136,"data_compressed":11794999,"data_compressed_allocated":561643520,"data_compressed_original":14039158784,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:23:04.221 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:04.221 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:04.221 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:04.222 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:04.222 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:23:24.223 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:23:27.636 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:23:27.650 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:23:27.669 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2238,"stamp":"2023-09-23T08:23:27.137076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20145821276,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1133176,"num_read_kb":4194596,"num_write":1060345,"num_write_kb":4183545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76795,"ondisk_log_size":76795,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15403192,"kb_used_data":12399964,"kb_used_omap":1738,"kb_used_meta":3001461,"kb_avail":267712328,"statfs":{"total":289910292480,"available":274137423872,"internally_reserved":0,"allocated":12697563136,"data_stored":54488165170,"data_compressed":36590661,"data_compressed_allocated":1685155840,"data_compressed_original":42079883264,"omap_allocated":1780229,"internal_metadata":3073496571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,15,11,10,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7491584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2140,"num_read_kb":8520,"num_write":2109,"num_write_kb":8436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007214"},"pg_stats":[{"pgid":"2.7","version":"96'5531","reported_seq":9292,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:23.448576+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:23:23.448576+0000","last_peered":"2023-09-23T08:23:23.448576+0000","last_clean":"2023-09-23T08:23:23.448576+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:23:23.448576+0000","last_undegraded":"2023-09-23T08:23:23.448576+0000","last_fullsized":"2023-09-23T08:23:23.448576+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'8082","reported_seq":13958,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:23.450055+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:23:23.450055+0000","last_peered":"2023-09-23T08:23:23.450055+0000","last_clean":"2023-09-23T08:23:23.450055+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:23:23.450055+0000","last_undegraded":"2023-09-23T08:23:23.450055+0000","last_fullsized":"2023-09-23T08:23:23.450055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3933","reported_seq":6454,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:23.447752+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:23:23.447752+0000","last_peered":"2023-09-23T08:23:23.447752+0000","last_clean":"2023-09-23T08:23:23.447752+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:23:23.447752+0000","last_undegraded":"2023-09-23T08:23:23.447752+0000","last_fullsized":"2023-09-23T08:23:23.447752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"96'267379","reported_seq":535207,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:24.634645+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:23:24.634645+0000","last_peered":"2023-09-23T08:23:24.634645+0000","last_clean":"2023-09-23T08:23:24.634645+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:23:24.634645+0000","last_undegraded":"2023-09-23T08:23:24.634645+0000","last_fullsized":"2023-09-23T08:23:24.634645+0000","mapping_epoch":18,"log_start":"94'257331","ondisk_log_start":"94'257331","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1493,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 268s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5149634560,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":268866,"num_read_kb":1053684,"num_write":265997,"num_write_kb":1063988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3042,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:17.307480+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:23:17.307480+0000","last_peered":"2023-09-23T08:23:17.307480+0000","last_clean":"2023-09-23T08:23:17.307480+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:23:17.307480+0000","last_undegraded":"2023-09-23T08:23:17.307480+0000","last_fullsized":"2023-09-23T08:23:17.307480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'252752","reported_seq":506231,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:24.123622+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:23:24.123622+0000","last_peered":"2023-09-23T08:23:24.123622+0000","last_clean":"2023-09-23T08:23:24.123622+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:23:24.123622+0000","last_undegraded":"2023-09-23T08:23:24.123622+0000","last_fullsized":"2023-09-23T08:23:24.123622+0000","mapping_epoch":18,"log_start":"94'242710","ondisk_log_start":"94'242710","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4879847443,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":254594,"num_read_kb":998180,"num_write":251451,"num_write_kb":1005798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":267,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:05.613517+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:22:05.613517+0000","last_peered":"2023-09-23T08:22:05.613517+0000","last_clean":"2023-09-23T08:22:05.613517+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:22:05.613517+0000","last_undegraded":"2023-09-23T08:22:05.613517+0000","last_fullsized":"2023-09-23T08:22:05.613517+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'6689","reported_seq":10392,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:23.488950+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:23:23.488950+0000","last_peered":"2023-09-23T08:23:23.488950+0000","last_clean":"2023-09-23T08:23:23.488950+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:23:23.488950+0000","last_undegraded":"2023-09-23T08:23:23.488950+0000","last_fullsized":"2023-09-23T08:23:23.488950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"96'254300","reported_seq":509878,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:24.114515+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:23:24.114515+0000","last_peered":"2023-09-23T08:23:24.114515+0000","last_clean":"2023-09-23T08:23:24.114515+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:23:24.114515+0000","last_undegraded":"2023-09-23T08:23:24.114515+0000","last_fullsized":"2023-09-23T08:23:24.114515+0000","mapping_epoch":18,"log_start":"94'244222","ondisk_log_start":"94'244222","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1899,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 257s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4896272384,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":256517,"num_read_kb":1005664,"num_write":253008,"num_write_kb":1012032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3674,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:12.169328+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:23:12.169328+0000","last_peered":"2023-09-23T08:23:12.169328+0000","last_clean":"2023-09-23T08:23:12.169328+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:23:12.169328+0000","last_undegraded":"2023-09-23T08:23:12.169328+0000","last_fullsized":"2023-09-23T08:23:12.169328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"96'271548","reported_seq":543195,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:24.660879+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:23:24.660879+0000","last_peered":"2023-09-23T08:23:24.660879+0000","last_clean":"2023-09-23T08:23:24.660879+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:23:24.660879+0000","last_undegraded":"2023-09-23T08:23:24.660879+0000","last_fullsized":"2023-09-23T08:23:24.660879+0000","mapping_epoch":18,"log_start":"94'261523","ondisk_log_start":"94'261523","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1532,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 272s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5219467264,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":272663,"num_read_kb":1069304,"num_write":270191,"num_write_kb":1080764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'4710","reported_seq":7212,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:22.416924+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:23:22.416924+0000","last_peered":"2023-09-23T08:23:22.416924+0000","last_clean":"2023-09-23T08:23:22.416924+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:23:22.416924+0000","last_undegraded":"2023-09-23T08:23:22.416924+0000","last_fullsized":"2023-09-23T08:23:22.416924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"96'4693","reported_seq":8981,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:23:23.447717+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:23:23.447717+0000","last_peered":"2023-09-23T08:23:23.447717+0000","last_clean":"2023-09-23T08:23:23.447717+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:23:23.447717+0000","last_undegraded":"2023-09-23T08:23:23.447717+0000","last_fullsized":"2023-09-23T08:23:23.447717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20145221651,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1052640,"num_read_kb":4126832,"num_write":1040647,"num_write_kb":4162582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12693843968,"data_stored":54485803008,"data_compressed":36585042,"data_compressed_allocated":1685082112,"data_compressed_original":42078527488,"omap_allocated":0,"internal_metadata":0},"log_size":40193,"ondisk_log_size":40193,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36503,"ondisk_log_size":36503,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072896,"kb_used_data":4132280,"kb_used_omap":573,"kb_used_meta":940034,"kb_avail":89298944,"statfs":{"total":96636764160,"available":91442118656,"internally_reserved":0,"allocated":4231454720,"data_stored":18162329182,"data_compressed":11794032,"data_compressed_allocated":561696768,"data_compressed_original":14027112448,"omap_allocated":587472,"internal_metadata":962595120},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":1.702,"15min":2.0790000000000002},"min":{"1min":0.28699999999999998,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":9.9939999999999998,"5min":11.561,"15min":13.019},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.4119999999999999,"5min":1.756,"15min":2.2749999999999999},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.185},"max":{"1min":10.034000000000001,"5min":11.597,"15min":22.266999999999999},"last":0.38700000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:22:34 2023","interfaces":[{"interface":"back","average":{"1min":3.4260000000000002,"5min":3.0630000000000002,"15min":3.198},"min":{"1min":0.216,"5min":0.215,"15min":0.186},"max":{"1min":14.371,"5min":16.27,"15min":23.045999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.8620000000000001,"5min":2.5369999999999999,"15min":2.8650000000000002},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.916,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739222,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5146976,"kb_used_data":4133908,"kb_used_omap":580,"kb_used_meta":1012475,"kb_avail":89224864,"statfs":{"total":96636764160,"available":91366260736,"internally_reserved":0,"allocated":4233121792,"data_stored":18162891179,"data_compressed":12387787,"data_compressed_allocated":561729536,"data_compressed_original":14026182656,"omap_allocated":594767,"internal_metadata":1036774577},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2,4,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.571,"15min":2.2090000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.5709999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":1.643},{"interface":"front","average":{"1min":2.411,"5min":2.7730000000000001,"15min":2.9329999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":14.401,"5min":15.093,"15min":24.385000000000002},"last":2.226}]},{"osd":2,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":2.9350000000000001,"5min":3.1749999999999998,"15min":3.4329999999999998},"min":{"1min":0.30499999999999999,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":8.468,"5min":15.138,"15min":25.84},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.742,"5min":2.4670000000000001,"15min":2.7069999999999999},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.3610000000000007,"5min":13.435,"15min":25.120999999999999},"last":0.42999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739221,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5183320,"kb_used_data":4133776,"kb_used_omap":583,"kb_used_meta":1048952,"kb_avail":89188520,"statfs":{"total":96636764160,"available":91329044480,"internally_reserved":0,"allocated":4232986624,"data_stored":18162944809,"data_compressed":12408842,"data_compressed_allocated":561729536,"data_compressed_original":14026588160,"omap_allocated":597990,"internal_metadata":1074126874},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,5,3,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.3159999999999998,"15min":2.6160000000000001},"min":{"1min":0.26800000000000002,"5min":0.17399999999999999,"15min":0.14699999999999999},"max":{"1min":8.6969999999999992,"5min":11.404999999999999,"15min":14.496},"last":1.71},{"interface":"front","average":{"1min":2.6459999999999999,"5min":2.9100000000000001,"15min":3.306},"min":{"1min":0.24299999999999999,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":12.968999999999999,"5min":15.09,"15min":16.693000000000001},"last":1.7569999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:22:27 2023","interfaces":[{"interface":"back","average":{"1min":3.327,"5min":3.843,"15min":3.367},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":11.367000000000001,"5min":18.414000000000001,"15min":20.216999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.365,"5min":1.72,"15min":1.7969999999999999},"min":{"1min":0.26300000000000001,"5min":0.218,"15min":0.187},"max":{"1min":7.4359999999999999,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4231925760,"data_stored":18161954816,"data_compressed":12385132,"data_compressed_allocated":561692672,"data_compressed_original":14025580544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4231602176,"data_stored":18161917952,"data_compressed":12405842,"data_compressed_allocated":561692672,"data_compressed_original":14025846784,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4230316032,"data_stored":18161930240,"data_compressed":11794068,"data_compressed_allocated":561696768,"data_compressed_original":14027100160,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:23:27.671 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:27.671 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:27.671 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:27.671 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:27.672 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-09-23T08:23:28.389 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-09-23T08:23:28.403 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-09-23T08:23:29.138 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-09-23T08:23:29.160 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-09-23T08:23:29.844 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-09-23T08:23:29.862 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-09-23T08:23:30.577 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-09-23T08:23:30.589 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:23:50.591 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:23:51.413 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:23:51.442 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:23:51.460 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-09-23T08:23:51.162605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20162995804,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1138368,"num_read_kb":4215264,"num_write":1065249,"num_write_kb":4203161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76933,"ondisk_log_size":76933,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15728848,"kb_used_data":12445176,"kb_used_omap":1738,"kb_used_meta":3281909,"kb_avail":267386672,"statfs":{"total":289910292480,"available":273803952128,"internally_reserved":0,"allocated":12743860224,"data_stored":54488558989,"data_compressed":36591638,"data_compressed_allocated":1685204992,"data_compressed_original":42036170752,"omap_allocated":1780229,"internal_metadata":3360675323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,11,17,26,12,6,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10543104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3187,"num_read_kb":12692,"num_write":3008,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.022261"},"pg_stats":[{"pgid":"2.7","version":"97'5552","reported_seq":9329,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.051846+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:23:48.051846+0000","last_peered":"2023-09-23T08:23:48.051846+0000","last_clean":"2023-09-23T08:23:48.051846+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:23:48.051846+0000","last_undegraded":"2023-09-23T08:23:48.051846+0000","last_fullsized":"2023-09-23T08:23:48.051846+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5552,"log_dups_size":0,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8110","reported_seq":14007,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.057734+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:23:48.057734+0000","last_peered":"2023-09-23T08:23:48.057734+0000","last_clean":"2023-09-23T08:23:48.057734+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:23:48.057734+0000","last_undegraded":"2023-09-23T08:23:48.057734+0000","last_fullsized":"2023-09-23T08:23:48.057734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'3947","reported_seq":6479,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.054890+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:23:48.054890+0000","last_peered":"2023-09-23T08:23:48.054890+0000","last_clean":"2023-09-23T08:23:48.054890+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:23:48.054890+0000","last_undegraded":"2023-09-23T08:23:48.054890+0000","last_fullsized":"2023-09-23T08:23:48.054890+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3947,"log_dups_size":0,"ondisk_log_size":3947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"97'268629","reported_seq":537813,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:49.994302+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:23:49.994302+0000","last_peered":"2023-09-23T08:23:49.994302+0000","last_clean":"2023-09-23T08:23:49.994302+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:23:49.994302+0000","last_undegraded":"2023-09-23T08:23:49.994302+0000","last_fullsized":"2023-09-23T08:23:49.994302+0000","mapping_epoch":18,"log_start":"94'258551","ondisk_log_start":"94'258551","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1644,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 293s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5153906688,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":270204,"num_read_kb":1059004,"num_write":267247,"num_write_kb":1068988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3053,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:42.932034+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:23:42.932034+0000","last_peered":"2023-09-23T08:23:42.932034+0000","last_clean":"2023-09-23T08:23:42.932034+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:23:42.932034+0000","last_undegraded":"2023-09-23T08:23:42.932034+0000","last_fullsized":"2023-09-23T08:23:42.932034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'253951","reported_seq":508693,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:49.716460+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:23:49.716460+0000","last_peered":"2023-09-23T08:23:49.716460+0000","last_clean":"2023-09-23T08:23:49.716460+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:23:49.716460+0000","last_undegraded":"2023-09-23T08:23:49.716460+0000","last_fullsized":"2023-09-23T08:23:49.716460+0000","mapping_epoch":18,"log_start":"94'243932","ondisk_log_start":"94'243932","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4884111379,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":255853,"num_read_kb":1003184,"num_write":252650,"num_write_kb":1010594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":268,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.359206+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:22:58.359206+0000","last_peered":"2023-09-23T08:22:58.359206+0000","last_clean":"2023-09-23T08:22:58.359206+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:22:58.359206+0000","last_undegraded":"2023-09-23T08:22:58.359206+0000","last_fullsized":"2023-09-23T08:22:58.359206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'6710","reported_seq":10430,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.124382+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:23:48.124382+0000","last_peered":"2023-09-23T08:23:48.124382+0000","last_clean":"2023-09-23T08:23:48.124382+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:23:48.124382+0000","last_undegraded":"2023-09-23T08:23:48.124382+0000","last_fullsized":"2023-09-23T08:23:48.124382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"97'255493","reported_seq":512370,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:49.694974+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:23:49.694974+0000","last_peered":"2023-09-23T08:23:49.694974+0000","last_clean":"2023-09-23T08:23:49.694974+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:23:49.694974+0000","last_undegraded":"2023-09-23T08:23:49.694974+0000","last_fullsized":"2023-09-23T08:23:49.694974+0000","mapping_epoch":18,"log_start":"94'245447","ondisk_log_start":"94'245447","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2082,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 283s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4900364288,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":257797,"num_read_kb":1010760,"num_write":254201,"num_write_kb":1016804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3680,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:37.805559+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:23:37.805559+0000","last_peered":"2023-09-23T08:23:37.805559+0000","last_clean":"2023-09-23T08:23:37.805559+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:23:37.805559+0000","last_undegraded":"2023-09-23T08:23:37.805559+0000","last_fullsized":"2023-09-23T08:23:37.805559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"97'272810","reported_seq":545788,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:23:50.066745+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:23:50.066745+0000","last_peered":"2023-09-23T08:23:50.066745+0000","last_clean":"2023-09-23T08:23:50.066745+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:23:50.066745+0000","last_undegraded":"2023-09-23T08:23:50.066745+0000","last_fullsized":"2023-09-23T08:23:50.066745+0000","mapping_epoch":18,"log_start":"94'262744","ondisk_log_start":"94'262744","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1710,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 297s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5224013824,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":273978,"num_read_kb":1074552,"num_write":271453,"num_write_kb":1085812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'4727","reported_seq":7240,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.012731+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:23:48.012731+0000","last_peered":"2023-09-23T08:23:48.012731+0000","last_clean":"2023-09-23T08:23:48.012731+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:23:48.012731+0000","last_undegraded":"2023-09-23T08:23:48.012731+0000","last_fullsized":"2023-09-23T08:23:48.012731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"97'4714","reported_seq":9019,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:48.047120+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:23:48.047120+0000","last_peered":"2023-09-23T08:23:48.047120+0000","last_clean":"2023-09-23T08:23:48.047120+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:23:48.047120+0000","last_undegraded":"2023-09-23T08:23:48.047120+0000","last_fullsized":"2023-09-23T08:23:48.047120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20162396179,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1057832,"num_read_kb":4147500,"num_write":1045551,"num_write_kb":4182198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12740345856,"data_stored":54486188032,"data_compressed":36585660,"data_compressed_allocated":1685143552,"data_compressed_original":42034622464,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36625,"ondisk_log_size":36625,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739229,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5173896,"kb_used_data":4147200,"kb_used_omap":573,"kb_used_meta":1026114,"kb_avail":89197944,"statfs":{"total":96636764160,"available":91338694656,"internally_reserved":0,"allocated":4246732800,"data_stored":18162460455,"data_compressed":11793828,"data_compressed_allocated":561704960,"data_compressed_original":14012678144,"omap_allocated":587472,"internal_metadata":1050741040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.702,"15min":1.976},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.1710000000000003,"5min":11.561,"15min":13.019},"last":5.7729999999999997},{"interface":"front","average":{"1min":1.244,"5min":1.748,"15min":2.1779999999999999},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":6.1950000000000003,"5min":11.597,"15min":22.266999999999999},"last":5.8129999999999997}]},{"osd":1,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":2.8980000000000001,"15min":3.1859999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.8409999999999993,"5min":16.27,"15min":23.045999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":2.129,"5min":2.4199999999999999,"15min":2.835},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":6.1769999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5266144,"kb_used_data":4148952,"kb_used_omap":580,"kb_used_meta":1116603,"kb_avail":89105696,"statfs":{"total":96636764160,"available":91244232704,"internally_reserved":0,"allocated":4248526848,"data_stored":18163075700,"data_compressed":12388885,"data_compressed_allocated":561750016,"data_compressed_original":14011695104,"omap_allocated":594767,"internal_metadata":1143401649},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,10,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.571,"15min":2.2090000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.5709999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":7.0190000000000001},{"interface":"front","average":{"1min":2.411,"5min":2.7730000000000001,"15min":2.9329999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":14.401,"5min":15.093,"15min":24.385000000000002},"last":5.923}]},{"osd":2,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":2.9350000000000001,"5min":3.1749999999999998,"15min":3.4329999999999998},"min":{"1min":0.30499999999999999,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":8.468,"5min":15.138,"15min":25.84},"last":7.1440000000000001},{"interface":"front","average":{"1min":2.742,"5min":2.4670000000000001,"15min":2.7069999999999999},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.3610000000000007,"5min":13.435,"15min":25.120999999999999},"last":7.4080000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739226,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5288808,"kb_used_data":4149024,"kb_used_omap":583,"kb_used_meta":1139192,"kb_avail":89083032,"statfs":{"total":96636764160,"available":91221024768,"internally_reserved":0,"allocated":4248600576,"data_stored":18163022834,"data_compressed":12408925,"data_compressed_allocated":561750016,"data_compressed_original":14011797504,"omap_allocated":597990,"internal_metadata":1166532634},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,10,4,10,2,5,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.2989999999999999,"5min":2.2730000000000001,"15min":2.6970000000000001},"min":{"1min":0.188,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":9.8979999999999997,"5min":11.404999999999999,"15min":14.496},"last":3.3039999999999998},{"interface":"front","average":{"1min":2.4209999999999998,"5min":2.77,"15min":3.258},"min":{"1min":0.33400000000000002,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":9.8409999999999993,"5min":15.09,"15min":16.693000000000001},"last":3.923}]},{"osd":2,"last update":"Sat Sep 23 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":3.9620000000000002,"15min":3.4129999999999998},"min":{"1min":0.27700000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":18.943999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":10.961},{"interface":"front","average":{"1min":1.853,"5min":1.756,"15min":1.8140000000000001},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.193},"max":{"1min":11.871,"5min":18.353000000000002,"15min":18.353000000000002},"last":10.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4247396352,"data_stored":18162065408,"data_compressed":12385859,"data_compressed_allocated":561717248,"data_compressed_original":14010957824,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4247187456,"data_stored":18162065408,"data_compressed":12406371,"data_compressed_allocated":561713152,"data_compressed_original":14011154432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4245762048,"data_stored":18162057216,"data_compressed":11793430,"data_compressed_allocated":561713152,"data_compressed_original":14012510208,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:23:51.462 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:51.462 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:51.462 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:51.462 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:23:51.463 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:24:11.463 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:24:12.179 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:24:12.193 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:24:12.213 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2260,"stamp":"2023-09-23T08:24:11.169151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20177241692,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1142376,"num_read_kb":4231208,"num_write":1069261,"num_write_kb":4219209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77063,"ondisk_log_size":77063,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15879236,"kb_used_data":12481836,"kb_used_omap":1738,"kb_used_meta":3395637,"kb_avail":267236284,"statfs":{"total":289910292480,"available":273649954816,"internally_reserved":0,"allocated":12781400064,"data_stored":54488993165,"data_compressed":36592011,"data_compressed_allocated":1685303296,"data_compressed_original":42000699392,"omap_allocated":1780229,"internal_metadata":3477132795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,4,15,29,24,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10694656,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2942,"num_read_kb":11712,"num_write":3015,"num_write_kb":12060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004639"},"pg_stats":[{"pgid":"2.7","version":"97'5567","reported_seq":9357,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.544608+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:24:08.544608+0000","last_peered":"2023-09-23T08:24:08.544608+0000","last_clean":"2023-09-23T08:24:08.544608+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:24:08.544608+0000","last_undegraded":"2023-09-23T08:24:08.544608+0000","last_fullsized":"2023-09-23T08:24:08.544608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8130","reported_seq":14044,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.569144+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:24:08.569144+0000","last_peered":"2023-09-23T08:24:08.569144+0000","last_clean":"2023-09-23T08:24:08.569144+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:24:08.569144+0000","last_undegraded":"2023-09-23T08:24:08.569144+0000","last_fullsized":"2023-09-23T08:24:08.569144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'3957","reported_seq":6498,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.531208+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:24:08.531208+0000","last_peered":"2023-09-23T08:24:08.531208+0000","last_clean":"2023-09-23T08:24:08.531208+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:24:08.531208+0000","last_undegraded":"2023-09-23T08:24:08.531208+0000","last_fullsized":"2023-09-23T08:24:08.531208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"97'269659","reported_seq":539869,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:10.590726+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:24:10.590726+0000","last_peered":"2023-09-23T08:24:10.590726+0000","last_clean":"2023-09-23T08:24:10.590726+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:24:10.590726+0000","last_undegraded":"2023-09-23T08:24:10.590726+0000","last_fullsized":"2023-09-23T08:24:10.590726+0000","mapping_epoch":18,"log_start":"94'259560","ondisk_log_start":"94'259560","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1789,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 314s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5157629952,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":271221,"num_read_kb":1063056,"num_write":268277,"num_write_kb":1073108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3061,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:03.440696+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:24:03.440696+0000","last_peered":"2023-09-23T08:24:03.440696+0000","last_clean":"2023-09-23T08:24:03.440696+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:24:03.440696+0000","last_undegraded":"2023-09-23T08:24:03.440696+0000","last_fullsized":"2023-09-23T08:24:03.440696+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'254913","reported_seq":510573,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:10.072989+0000","last_change":"2023-09-23T07:35:03.042951+0000","last_active":"2023-09-23T08:24:10.072989+0000","last_peered":"2023-09-23T08:24:10.072989+0000","last_clean":"2023-09-23T08:24:10.072989+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:24:10.072989+0000","last_undegraded":"2023-09-23T08:24:10.072989+0000","last_fullsized":"2023-09-23T08:24:10.072989+0000","mapping_epoch":18,"log_start":"94'244845","ondisk_log_start":"94'244845","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4887552019,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":256772,"num_read_kb":1006828,"num_write":253612,"num_write_kb":1014442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":268,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.359206+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:22:58.359206+0000","last_peered":"2023-09-23T08:22:58.359206+0000","last_clean":"2023-09-23T08:22:58.359206+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:22:58.359206+0000","last_undegraded":"2023-09-23T08:22:58.359206+0000","last_fullsized":"2023-09-23T08:22:58.359206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'6725","reported_seq":10454,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.665019+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:24:08.665019+0000","last_peered":"2023-09-23T08:24:08.665019+0000","last_clean":"2023-09-23T08:24:08.665019+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:24:08.665019+0000","last_undegraded":"2023-09-23T08:24:08.665019+0000","last_fullsized":"2023-09-23T08:24:08.665019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"97'256503","reported_seq":514376,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:10.144662+0000","last_change":"2023-09-23T08:19:06.385244+0000","last_active":"2023-09-23T08:24:10.144662+0000","last_peered":"2023-09-23T08:24:10.144662+0000","last_clean":"2023-09-23T08:24:10.144662+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:24:10.144662+0000","last_undegraded":"2023-09-23T08:24:10.144662+0000","last_fullsized":"2023-09-23T08:24:10.144662+0000","mapping_epoch":18,"log_start":"94'246469","ondisk_log_start":"94'246469","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2186,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4903874560,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":258780,"num_read_kb":1014672,"num_write":255211,"num_write_kb":1020844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3684,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:58.249641+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:23:58.249641+0000","last_peered":"2023-09-23T08:23:58.249641+0000","last_clean":"2023-09-23T08:23:58.249641+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:23:58.249641+0000","last_undegraded":"2023-09-23T08:23:58.249641+0000","last_fullsized":"2023-09-23T08:23:58.249641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"97'273820","reported_seq":547900,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:10.577396+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:24:10.577396+0000","last_peered":"2023-09-23T08:24:10.577396+0000","last_clean":"2023-09-23T08:24:10.577396+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:24:10.577396+0000","last_undegraded":"2023-09-23T08:24:10.577396+0000","last_fullsized":"2023-09-23T08:24:10.577396+0000","mapping_epoch":18,"log_start":"94'263771","ondisk_log_start":"94'263771","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1818,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5227585536,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":275067,"num_read_kb":1078888,"num_write":272463,"num_write_kb":1089852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'4741","reported_seq":7262,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.492152+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:24:08.492152+0000","last_peered":"2023-09-23T08:24:08.492152+0000","last_clean":"2023-09-23T08:24:08.492152+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:24:08.492152+0000","last_undegraded":"2023-09-23T08:24:08.492152+0000","last_fullsized":"2023-09-23T08:24:08.492152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"97'4729","reported_seq":9046,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:08.544385+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:24:08.544385+0000","last_peered":"2023-09-23T08:24:08.544385+0000","last_clean":"2023-09-23T08:24:08.544385+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:24:08.544385+0000","last_undegraded":"2023-09-23T08:24:08.544385+0000","last_fullsized":"2023-09-23T08:24:08.544385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20176642067,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1061840,"num_read_kb":4163444,"num_write":1049563,"num_write_kb":4198246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12778528768,"data_stored":54486708224,"data_compressed":36586624,"data_compressed_allocated":1685229568,"data_compressed_original":41998598144,"omap_allocated":0,"internal_metadata":0},"log_size":40250,"ondisk_log_size":40250,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36714,"ondisk_log_size":36714,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5218044,"kb_used_data":4159412,"kb_used_omap":573,"kb_used_meta":1058050,"kb_avail":89153796,"statfs":{"total":96636764160,"available":91293487104,"internally_reserved":0,"allocated":4259237888,"data_stored":18162616103,"data_compressed":11792718,"data_compressed_allocated":561745920,"data_compressed_original":14000877568,"omap_allocated":587472,"internal_metadata":1083443504},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.702,"15min":1.976},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.1710000000000003,"5min":11.561,"15min":13.019},"last":4.766},{"interface":"front","average":{"1min":1.244,"5min":1.748,"15min":2.1779999999999999},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":6.1950000000000003,"5min":11.597,"15min":22.266999999999999},"last":4.915}]},{"osd":1,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":2.8980000000000001,"15min":3.1859999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.8409999999999993,"5min":16.27,"15min":23.045999999999999},"last":4.6929999999999996},{"interface":"front","average":{"1min":2.129,"5min":2.4199999999999999,"15min":2.835},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":6.1769999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":3.3180000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739231,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5310176,"kb_used_data":4161112,"kb_used_omap":580,"kb_used_meta":1148475,"kb_avail":89061664,"statfs":{"total":96636764160,"available":91199143936,"internally_reserved":0,"allocated":4260978688,"data_stored":18163186292,"data_compressed":12389463,"data_compressed_allocated":561778688,"data_compressed_original":13999894528,"omap_allocated":594767,"internal_metadata":1176038577},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,19,19,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.571,"15min":2.2090000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.5709999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":7.3579999999999997},{"interface":"front","average":{"1min":2.411,"5min":2.7730000000000001,"15min":2.9329999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":14.401,"5min":15.093,"15min":24.385000000000002},"last":8.3279999999999994}]},{"osd":2,"last update":"Sat Sep 23 08:23:15 2023","interfaces":[{"interface":"back","average":{"1min":2.9350000000000001,"5min":3.1749999999999998,"15min":3.4329999999999998},"min":{"1min":0.30499999999999999,"5min":0.218,"15min":0.19500000000000001},"max":{"1min":8.468,"5min":15.138,"15min":25.84},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.742,"5min":2.4670000000000001,"15min":2.7069999999999999},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.3610000000000007,"5min":13.435,"15min":25.120999999999999},"last":7.4100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5351016,"kb_used_data":4161312,"kb_used_omap":583,"kb_used_meta":1189112,"kb_avail":89020824,"statfs":{"total":96636764160,"available":91157323776,"internally_reserved":0,"allocated":4261183488,"data_stored":18163190770,"data_compressed":12409830,"data_compressed_allocated":561778688,"data_compressed_original":13999927296,"omap_allocated":597990,"internal_metadata":1217650714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,3,3,3,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":2.2989999999999999,"5min":2.2730000000000001,"15min":2.6970000000000001},"min":{"1min":0.188,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":9.8979999999999997,"5min":11.404999999999999,"15min":14.496},"last":1.331},{"interface":"front","average":{"1min":2.4209999999999998,"5min":2.77,"15min":3.258},"min":{"1min":0.33400000000000002,"5min":0.20300000000000001,"15min":0.192},"max":{"1min":9.8409999999999993,"5min":15.09,"15min":16.693000000000001},"last":1.077}]},{"osd":2,"last update":"Sat Sep 23 08:23:27 2023","interfaces":[{"interface":"back","average":{"1min":3.282,"5min":3.9620000000000002,"15min":3.4129999999999998},"min":{"1min":0.27700000000000002,"5min":0.20300000000000001,"15min":0.19},"max":{"1min":18.943999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":4.6349999999999998},{"interface":"front","average":{"1min":1.853,"5min":1.756,"15min":1.8140000000000001},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.193},"max":{"1min":11.871,"5min":18.353000000000002,"15min":18.353000000000002},"last":4.6520000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4260167680,"data_stored":18162225152,"data_compressed":12386747,"data_compressed_allocated":561741824,"data_compressed_original":13998899200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4259897344,"data_stored":18162253824,"data_compressed":12407195,"data_compressed_allocated":561741824,"data_compressed_original":13999177728,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4258463744,"data_stored":18162229248,"data_compressed":11792682,"data_compressed_allocated":561745920,"data_compressed_original":14000521216,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:24:12.215 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:12.215 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:12.215 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:12.215 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:12.216 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:24:32.216 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:24:32.919 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:24:32.925 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:24:32.938 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2023-09-23T08:24:31.186415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20191385180,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1146233,"num_read_kb":4246512,"num_write":1073295,"num_write_kb":4235345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77070,"ondisk_log_size":77070,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16183628,"kb_used_data":12518316,"kb_used_omap":1738,"kb_used_meta":3663541,"kb_avail":266931892,"statfs":{"total":289910292480,"available":273338257408,"internally_reserved":0,"allocated":12818755584,"data_stored":54489476493,"data_compressed":36592980,"data_compressed_allocated":1685385216,"data_compressed_original":41965629440,"omap_allocated":1780229,"internal_metadata":3751466491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,11,14,24,22,21,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10592256,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2874,"num_read_kb":11400,"num_write":3022,"num_write_kb":12088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010675"},"pg_stats":[{"pgid":"2.7","version":"97'5588","reported_seq":9390,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:29.357722+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:24:29.357722+0000","last_peered":"2023-09-23T08:24:29.357722+0000","last_clean":"2023-09-23T08:24:29.357722+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:24:29.357722+0000","last_undegraded":"2023-09-23T08:24:29.357722+0000","last_fullsized":"2023-09-23T08:24:29.357722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'8158","reported_seq":14088,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:29.364365+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:24:29.364365+0000","last_peered":"2023-09-23T08:24:29.364365+0000","last_clean":"2023-09-23T08:24:29.364365+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:24:29.364365+0000","last_undegraded":"2023-09-23T08:24:29.364365+0000","last_fullsized":"2023-09-23T08:24:29.364365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'3971","reported_seq":6520,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:29.356688+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:24:29.356688+0000","last_peered":"2023-09-23T08:24:29.356688+0000","last_clean":"2023-09-23T08:24:29.356688+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:24:29.356688+0000","last_undegraded":"2023-09-23T08:24:29.356688+0000","last_fullsized":"2023-09-23T08:24:29.356688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"97'270668","reported_seq":541876,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:31.061114+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:24:31.061114+0000","last_peered":"2023-09-23T08:24:31.061114+0000","last_clean":"2023-09-23T08:24:31.061114+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:24:31.061114+0000","last_undegraded":"2023-09-23T08:24:31.061114+0000","last_fullsized":"2023-09-23T08:24:31.061114+0000","mapping_epoch":18,"log_start":"95'260571","ondisk_log_start":"95'260571","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1936,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 334s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5161267200,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":272203,"num_read_kb":1066948,"num_write":269286,"num_write_kb":1077144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3069,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:23.847506+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:24:23.847506+0000","last_peered":"2023-09-23T08:24:23.847506+0000","last_clean":"2023-09-23T08:24:23.847506+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:24:23.847506+0000","last_undegraded":"2023-09-23T08:24:23.847506+0000","last_fullsized":"2023-09-23T08:24:23.847506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'255866","reported_seq":512484,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:30.462636+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:24:30.462636+0000","last_peered":"2023-09-23T08:24:30.462636+0000","last_clean":"2023-09-23T08:24:30.462636+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:24:30.462636+0000","last_undegraded":"2023-09-23T08:24:30.462636+0000","last_fullsized":"2023-09-23T08:24:30.462636+0000","mapping_epoch":18,"log_start":"95'245860","ondisk_log_start":"95'245860","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":86,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4890927123,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":257721,"num_read_kb":1010600,"num_write":254565,"num_write_kb":1018254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":268,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-09-23T08:22:58.359206+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:22:58.359206+0000","last_peered":"2023-09-23T08:22:58.359206+0000","last_clean":"2023-09-23T08:22:58.359206+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:22:58.359206+0000","last_undegraded":"2023-09-23T08:22:58.359206+0000","last_fullsized":"2023-09-23T08:22:58.359206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'6746","reported_seq":10487,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:29.356379+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:24:29.356379+0000","last_peered":"2023-09-23T08:24:29.356379+0000","last_clean":"2023-09-23T08:24:29.356379+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:24:29.356379+0000","last_undegraded":"2023-09-23T08:24:29.356379+0000","last_fullsized":"2023-09-23T08:24:29.356379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"97'257480","reported_seq":516297,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:30.418419+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:24:30.418419+0000","last_peered":"2023-09-23T08:24:30.418419+0000","last_clean":"2023-09-23T08:24:30.418419+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:24:30.418419+0000","last_undegraded":"2023-09-23T08:24:30.418419+0000","last_fullsized":"2023-09-23T08:24:30.418419+0000","mapping_epoch":18,"log_start":"95'247476","ondisk_log_start":"95'247476","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4907155456,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":259714,"num_read_kb":1018372,"num_write":256188,"num_write_kb":1024752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3688,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:18.731189+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:24:18.731189+0000","last_peered":"2023-09-23T08:24:18.731189+0000","last_clean":"2023-09-23T08:24:18.731189+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:24:18.731189+0000","last_undegraded":"2023-09-23T08:24:18.731189+0000","last_fullsized":"2023-09-23T08:24:18.731189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"97'274915","reported_seq":550002,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:31.088117+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:24:31.088117+0000","last_peered":"2023-09-23T08:24:31.088117+0000","last_clean":"2023-09-23T08:24:31.088117+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:24:31.088117+0000","last_undegraded":"2023-09-23T08:24:31.088117+0000","last_fullsized":"2023-09-23T08:24:31.088117+0000","mapping_epoch":18,"log_start":"95'264884","ondisk_log_start":"95'264884","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1908,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 338s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5231435776,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":276059,"num_read_kb":1082828,"num_write":273558,"num_write_kb":1094232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'4755","reported_seq":7284,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:30.906020+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:24:30.906020+0000","last_peered":"2023-09-23T08:24:30.906020+0000","last_clean":"2023-09-23T08:24:30.906020+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:24:30.906020+0000","last_undegraded":"2023-09-23T08:24:30.906020+0000","last_fullsized":"2023-09-23T08:24:30.906020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"97'4750","reported_seq":9079,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:24:29.357492+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:24:29.357492+0000","last_peered":"2023-09-23T08:24:29.357492+0000","last_clean":"2023-09-23T08:24:29.357492+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:24:29.357492+0000","last_undegraded":"2023-09-23T08:24:29.357492+0000","last_fullsized":"2023-09-23T08:24:29.357492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20190785555,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1065697,"num_read_kb":4178748,"num_write":1053597,"num_write_kb":4214382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12816355328,"data_stored":54487085056,"data_compressed":36587298,"data_compressed_allocated":1685303296,"data_compressed_original":41962934272,"omap_allocated":0,"internal_metadata":0},"log_size":40138,"ondisk_log_size":40138,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36833,"ondisk_log_size":36833,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739237,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5314760,"kb_used_data":4171452,"kb_used_omap":573,"kb_used_meta":1142722,"kb_avail":89057080,"statfs":{"total":96636764160,"available":91194449920,"internally_reserved":0,"allocated":4271566848,"data_stored":18162763559,"data_compressed":11792346,"data_compressed_allocated":561770496,"data_compressed_original":13989298176,"omap_allocated":587472,"internal_metadata":1170147632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,7,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.702,"15min":1.976},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.1710000000000003,"5min":11.561,"15min":13.019},"last":7.2800000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.748,"15min":2.1779999999999999},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":6.1950000000000003,"5min":11.597,"15min":22.266999999999999},"last":0.215}]},{"osd":1,"last update":"Sat Sep 23 08:23:34 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":2.8980000000000001,"15min":3.1859999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.8409999999999993,"5min":16.27,"15min":23.045999999999999},"last":2.319},{"interface":"front","average":{"1min":2.129,"5min":2.4199999999999999,"15min":2.835},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":6.1769999999999996,"5min":28.257000000000001,"15min":28.257000000000001},"last":0.32000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739235,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5425216,"kb_used_data":4173428,"kb_used_omap":580,"kb_used_meta":1251195,"kb_avail":88946624,"statfs":{"total":96636764160,"available":91081342976,"internally_reserved":0,"allocated":4273590272,"data_stored":18163366516,"data_compressed":12390096,"data_compressed_allocated":561807360,"data_compressed_original":13988065280,"omap_allocated":594767,"internal_metadata":1281223857},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,4,7,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.415,"15min":2.2469999999999999},"min":{"1min":0.191,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.3579999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":2.3700000000000001},{"interface":"front","average":{"1min":3.1749999999999998,"5min":2.9079999999999999,"15min":3.069},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.507999999999999,"5min":15.093,"15min":24.385000000000002},"last":1.377}]},{"osd":2,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":3.0059999999999998,"15min":3.351},"min":{"1min":0.29999999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":7.1440000000000001,"5min":15.138,"15min":25.84},"last":1.0860000000000001},{"interface":"front","average":{"1min":2.2909999999999999,"5min":2.4060000000000001,"15min":2.7370000000000001},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.4100000000000001,"5min":12.609,"15min":25.120999999999999},"last":8.3480000000000008}]}]},{"osd":1,"up_from":8,"seq":34359739234,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5443652,"kb_used_data":4173436,"kb_used_omap":583,"kb_used_meta":1269624,"kb_avail":88928188,"statfs":{"total":96636764160,"available":91062464512,"internally_reserved":0,"allocated":4273598464,"data_stored":18163346418,"data_compressed":12410538,"data_compressed_allocated":561807360,"data_compressed_original":13988265984,"omap_allocated":597990,"internal_metadata":1300095002},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,13,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":4.0069999999999997,"5min":2.6960000000000002,"15min":2.7170000000000001},"min":{"1min":0.35399999999999998,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":29.236999999999998,"5min":29.236999999999998,"15min":29.236999999999998},"last":10.840999999999999},{"interface":"front","average":{"1min":5.1539999999999999,"5min":3.242,"15min":3.335},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":29.521000000000001,"5min":29.521000000000001,"15min":29.521000000000001},"last":10.99}]},{"osd":2,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.7789999999999999,"15min":3.419},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":16.527000000000001,"5min":18.943999999999999,"15min":20.216999999999999},"last":11.106999999999999},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.601,"15min":1.7769999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.193},"max":{"1min":10.224,"5min":11.871,"15min":18.353000000000002},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4272861184,"data_stored":18162368512,"data_compressed":12387331,"data_compressed_allocated":561766400,"data_compressed_original":13986930688,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4272484352,"data_stored":18162364416,"data_compressed":12407616,"data_compressed_allocated":561770496,"data_compressed_original":13987299328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4271009792,"data_stored":18162352128,"data_compressed":11792351,"data_compressed_allocated":561766400,"data_compressed_original":13988704256,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:24:32.940 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:32.940 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:32.940 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:32.941 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:24:52.942 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:24:53.720 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:24:53.736 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:24:53.759 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2282,"stamp":"2023-09-23T08:24:53.199977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20205299292,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1149996,"num_read_kb":4261472,"num_write":1077254,"num_write_kb":4251181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77170,"ondisk_log_size":77170,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16444140,"kb_used_data":12555540,"kb_used_omap":1738,"kb_used_meta":3886837,"kb_avail":266671380,"statfs":{"total":289910292480,"available":273071493120,"internally_reserved":0,"allocated":12856872960,"data_stored":54489677671,"data_compressed":36589513,"data_compressed_allocated":1685315584,"data_compressed_original":41928953856,"omap_allocated":1780229,"internal_metadata":3980121595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,7,6,3,9,16,27,18,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8708096,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2333,"num_read_kb":9284,"num_write":2470,"num_write_kb":9880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009224"},"pg_stats":[{"pgid":"2.7","version":"98'5603","reported_seq":9418,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:49.450316+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:24:49.450316+0000","last_peered":"2023-09-23T08:24:49.450316+0000","last_clean":"2023-09-23T08:24:49.450316+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:24:49.450316+0000","last_undegraded":"2023-09-23T08:24:49.450316+0000","last_fullsized":"2023-09-23T08:24:49.450316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8178","reported_seq":14125,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:49.452953+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:24:49.452953+0000","last_peered":"2023-09-23T08:24:49.452953+0000","last_clean":"2023-09-23T08:24:49.452953+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:24:49.452953+0000","last_undegraded":"2023-09-23T08:24:49.452953+0000","last_fullsized":"2023-09-23T08:24:49.452953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'3981","reported_seq":6539,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:49.450797+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:24:49.450797+0000","last_peered":"2023-09-23T08:24:49.450797+0000","last_clean":"2023-09-23T08:24:49.450797+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:24:49.450797+0000","last_undegraded":"2023-09-23T08:24:49.450797+0000","last_fullsized":"2023-09-23T08:24:49.450797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"98'271701","reported_seq":543924,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:51.551241+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:24:51.551241+0000","last_peered":"2023-09-23T08:24:51.551241+0000","last_clean":"2023-09-23T08:24:51.551241+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:24:51.551241+0000","last_undegraded":"2023-09-23T08:24:51.551241+0000","last_fullsized":"2023-09-23T08:24:51.551241+0000","mapping_epoch":18,"log_start":"95'261684","ondisk_log_start":"95'261684","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2054,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 355s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5164949504,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":273207,"num_read_kb":1070940,"num_write":270319,"num_write_kb":1081276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3079,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:44.331627+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:24:44.331627+0000","last_peered":"2023-09-23T08:24:44.331627+0000","last_clean":"2023-09-23T08:24:44.331627+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:24:44.331627+0000","last_undegraded":"2023-09-23T08:24:44.331627+0000","last_fullsized":"2023-09-23T08:24:44.331627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'256821","reported_seq":514364,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:50.946691+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:24:50.946691+0000","last_peered":"2023-09-23T08:24:50.946691+0000","last_clean":"2023-09-23T08:24:50.946691+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:24:50.946691+0000","last_undegraded":"2023-09-23T08:24:50.946691+0000","last_fullsized":"2023-09-23T08:24:50.946691+0000","mapping_epoch":18,"log_start":"95'246765","ondisk_log_start":"95'246765","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":284,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 30s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4894273555,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":258630,"num_read_kb":1014216,"num_write":255520,"num_write_kb":1022074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":269,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-09-23T08:23:25.663915+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:23:25.663915+0000","last_peered":"2023-09-23T08:23:25.663915+0000","last_clean":"2023-09-23T08:23:25.663915+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:23:25.663915+0000","last_undegraded":"2023-09-23T08:23:25.663915+0000","last_fullsized":"2023-09-23T08:23:25.663915+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'6761","reported_seq":10515,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:49.489665+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:24:49.489665+0000","last_peered":"2023-09-23T08:24:49.489665+0000","last_clean":"2023-09-23T08:24:49.489665+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:24:49.489665+0000","last_undegraded":"2023-09-23T08:24:49.489665+0000","last_fullsized":"2023-09-23T08:24:49.489665+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6761,"log_dups_size":0,"ondisk_log_size":6761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'258462","reported_seq":518178,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:50.929168+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:24:50.929168+0000","last_peered":"2023-09-23T08:24:50.929168+0000","last_clean":"2023-09-23T08:24:50.929168+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:24:50.929168+0000","last_undegraded":"2023-09-23T08:24:50.929168+0000","last_fullsized":"2023-09-23T08:24:50.929168+0000","mapping_epoch":18,"log_start":"95'248384","ondisk_log_start":"95'248384","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4910522368,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":260611,"num_read_kb":1021924,"num_write":257170,"num_write_kb":1028680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3693,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:39.331685+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:24:39.331685+0000","last_peered":"2023-09-23T08:24:39.331685+0000","last_clean":"2023-09-23T08:24:39.331685+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:24:39.331685+0000","last_undegraded":"2023-09-23T08:24:39.331685+0000","last_fullsized":"2023-09-23T08:24:39.331685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"98'275904","reported_seq":551954,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:24:51.536186+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:24:51.536186+0000","last_peered":"2023-09-23T08:24:51.536186+0000","last_clean":"2023-09-23T08:24:51.536186+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:24:51.536186+0000","last_undegraded":"2023-09-23T08:24:51.536186+0000","last_fullsized":"2023-09-23T08:24:51.536186+0000","mapping_epoch":18,"log_start":"95'265900","ondisk_log_start":"95'265900","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2019,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 359s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5234954240,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":277012,"num_read_kb":1086628,"num_write":274547,"num_write_kb":1098188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'4763","reported_seq":7301,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:51.476837+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:24:51.476837+0000","last_peered":"2023-09-23T08:24:51.476837+0000","last_clean":"2023-09-23T08:24:51.476837+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:24:51.476837+0000","last_undegraded":"2023-09-23T08:24:51.476837+0000","last_fullsized":"2023-09-23T08:24:51.476837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"98'4765","reported_seq":9107,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:49.449783+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:24:49.449783+0000","last_peered":"2023-09-23T08:24:49.449783+0000","last_clean":"2023-09-23T08:24:49.449783+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:24:49.449783+0000","last_undegraded":"2023-09-23T08:24:49.449783+0000","last_fullsized":"2023-09-23T08:24:49.449783+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20204699667,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1069460,"num_read_kb":4193708,"num_write":1057556,"num_write_kb":4230218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12854403072,"data_stored":54487293952,"data_compressed":36583785,"data_compressed_allocated":1685241856,"data_compressed_original":41926356992,"omap_allocated":0,"internal_metadata":0},"log_size":40155,"ondisk_log_size":40155,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":36916,"ondisk_log_size":36916,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5395576,"kb_used_data":4183856,"kb_used_omap":573,"kb_used_meta":1211138,"kb_avail":88976264,"statfs":{"total":96636764160,"available":91111694336,"internally_reserved":0,"allocated":4284268544,"data_stored":18162821061,"data_compressed":11790785,"data_compressed_allocated":561745920,"data_compressed_original":13977071616,"omap_allocated":587472,"internal_metadata":1240205616},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,4,3,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":1.976,"15min":2.0870000000000002},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":14.285,"5min":14.285,"15min":14.285},"last":5.4619999999999997},{"interface":"front","average":{"1min":2.8780000000000001,"5min":1.9430000000000001,"15min":2.246},"min":{"1min":0.215,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":14.321,"5min":14.321,"15min":22.266999999999999},"last":0.34200000000000003}]},{"osd":1,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":2.8639999999999999,"15min":3.097},"min":{"1min":0.26100000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.1839999999999993,"5min":16.27,"15min":23.045999999999999},"last":0.374},{"interface":"front","average":{"1min":2.5470000000000002,"5min":2.1989999999999998,"15min":2.7349999999999999},"min":{"1min":0.29499999999999998,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.081,"5min":12.044,"15min":28.257000000000001},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739239,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5524304,"kb_used_data":4186056,"kb_used_omap":580,"kb_used_meta":1337659,"kb_avail":88847536,"statfs":{"total":96636764160,"available":90979876864,"internally_reserved":0,"allocated":4286521344,"data_stored":18163415826,"data_compressed":12389340,"data_compressed_allocated":561782784,"data_compressed_original":13975605248,"omap_allocated":594767,"internal_metadata":1369762993},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,3,0,8,11,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.415,"15min":2.2469999999999999},"min":{"1min":0.191,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.3579999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.1749999999999998,"5min":2.9079999999999999,"15min":3.069},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.507999999999999,"5min":15.093,"15min":24.385000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":3.0059999999999998,"15min":3.351},"min":{"1min":0.29999999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":7.1440000000000001,"5min":15.138,"15min":25.84},"last":0.91900000000000004},{"interface":"front","average":{"1min":2.2909999999999999,"5min":2.4060000000000001,"15min":2.7370000000000001},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.4100000000000001,"5min":12.609,"15min":25.120999999999999},"last":0.97599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739238,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5524260,"kb_used_data":4185628,"kb_used_omap":583,"kb_used_meta":1338040,"kb_avail":88847580,"statfs":{"total":96636764160,"available":90979921920,"internally_reserved":0,"allocated":4286083072,"data_stored":18163440784,"data_compressed":12409388,"data_compressed_allocated":561786880,"data_compressed_original":13976276992,"omap_allocated":597990,"internal_metadata":1370152986},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,5,5,6,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":4.0069999999999997,"5min":2.6960000000000002,"15min":2.7170000000000001},"min":{"1min":0.35399999999999998,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":29.236999999999998,"5min":29.236999999999998,"15min":29.236999999999998},"last":10.467000000000001},{"interface":"front","average":{"1min":5.1539999999999999,"5min":3.242,"15min":3.335},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":29.521000000000001,"5min":29.521000000000001,"15min":29.521000000000001},"last":10.515000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.7789999999999999,"15min":3.419},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":16.527000000000001,"5min":18.943999999999999,"15min":20.216999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.601,"15min":1.7769999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.193},"max":{"1min":10.224,"5min":11.871,"15min":18.353000000000002},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4285538304,"data_stored":18162458624,"data_compressed":12386814,"data_compressed_allocated":561745920,"data_compressed_original":13974765568,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4285177856,"data_stored":18162421760,"data_compressed":12406294,"data_compressed_allocated":561745920,"data_compressed_original":13975080960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4283686912,"data_stored":18162413568,"data_compressed":11790677,"data_compressed_allocated":561750016,"data_compressed_original":13976510464,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:24:53.761 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:53.761 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:53.761 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:24:53.762 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:25:13.762 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:25:14.546 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:25:14.563 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:25:14.584 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2292,"stamp":"2023-09-23T08:25:13.207731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20219545180,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1153964,"num_read_kb":4277200,"num_write":1081243,"num_write_kb":4267137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77313,"ondisk_log_size":77313,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16649368,"kb_used_data":12592000,"kb_used_omap":1738,"kb_used_meta":4055605,"kb_avail":266466152,"statfs":{"total":289910292480,"available":272861339648,"internally_reserved":0,"allocated":12894208000,"data_stored":54490435431,"data_compressed":36589330,"data_compressed_allocated":1685295104,"data_compressed_original":41893777408,"omap_allocated":1780229,"internal_metadata":4152940027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,7,11,11,11,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9007104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2486,"num_read_kb":9856,"num_write":2515,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006276"},"pg_stats":[{"pgid":"2.7","version":"98'5622","reported_seq":9449,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:09.931703+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:25:09.931703+0000","last_peered":"2023-09-23T08:25:09.931703+0000","last_clean":"2023-09-23T08:25:09.931703+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:25:09.931703+0000","last_undegraded":"2023-09-23T08:25:09.931703+0000","last_fullsized":"2023-09-23T08:25:09.931703+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8202","reported_seq":14165,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:09.931694+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:25:09.931694+0000","last_peered":"2023-09-23T08:25:09.931694+0000","last_clean":"2023-09-23T08:25:09.931694+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:25:09.931694+0000","last_undegraded":"2023-09-23T08:25:09.931694+0000","last_fullsized":"2023-09-23T08:25:09.931694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'3993","reported_seq":6559,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:09.931480+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:25:09.931480+0000","last_peered":"2023-09-23T08:25:09.931480+0000","last_clean":"2023-09-23T08:25:09.931480+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:25:09.931480+0000","last_undegraded":"2023-09-23T08:25:09.931480+0000","last_fullsized":"2023-09-23T08:25:09.931480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"98'272731","reported_seq":545970,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:11.978989+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:25:11.978989+0000","last_peered":"2023-09-23T08:25:11.978989+0000","last_clean":"2023-09-23T08:25:11.978989+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:25:11.978989+0000","last_undegraded":"2023-09-23T08:25:11.978989+0000","last_fullsized":"2023-09-23T08:25:11.978989+0000","mapping_epoch":18,"log_start":"95'262708","ondisk_log_start":"95'262708","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2148,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 375s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5168660480,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":274212,"num_read_kb":1074920,"num_write":271349,"num_write_kb":1085396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3087,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:04.859038+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:25:04.859038+0000","last_peered":"2023-09-23T08:25:04.859038+0000","last_clean":"2023-09-23T08:25:04.859038+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:25:04.859038+0000","last_undegraded":"2023-09-23T08:25:04.859038+0000","last_fullsized":"2023-09-23T08:25:04.859038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'257760","reported_seq":516304,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:11.365963+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:25:11.365963+0000","last_peered":"2023-09-23T08:25:11.365963+0000","last_clean":"2023-09-23T08:25:11.365963+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:25:11.365963+0000","last_undegraded":"2023-09-23T08:25:11.365963+0000","last_fullsized":"2023-09-23T08:25:11.365963+0000","mapping_epoch":18,"log_start":"95'247678","ondisk_log_start":"95'247678","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":486,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 50s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4897726483,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":259619,"num_read_kb":1018132,"num_write":256459,"num_write_kb":1025830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":270,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:35.729748+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:24:35.729748+0000","last_peered":"2023-09-23T08:24:35.729748+0000","last_clean":"2023-09-23T08:24:35.729748+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:24:35.729748+0000","last_undegraded":"2023-09-23T08:24:35.729748+0000","last_fullsized":"2023-09-23T08:24:35.729748+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'6779","reported_seq":10545,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:09.945402+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:25:09.945402+0000","last_peered":"2023-09-23T08:25:09.945402+0000","last_clean":"2023-09-23T08:25:09.945402+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:25:09.945402+0000","last_undegraded":"2023-09-23T08:25:09.945402+0000","last_fullsized":"2023-09-23T08:25:09.945402+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'259441","reported_seq":520124,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:11.286262+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:25:11.286262+0000","last_peered":"2023-09-23T08:25:11.286262+0000","last_clean":"2023-09-23T08:25:11.286262+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:25:11.286262+0000","last_undegraded":"2023-09-23T08:25:11.286262+0000","last_fullsized":"2023-09-23T08:25:11.286262+0000","mapping_epoch":18,"log_start":"95'249387","ondisk_log_start":"95'249387","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4913971200,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":261580,"num_read_kb":1025776,"num_write":258149,"num_write_kb":1032596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3698,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:59.688433+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:24:59.688433+0000","last_peered":"2023-09-23T08:24:59.688433+0000","last_clean":"2023-09-23T08:24:59.688433+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:24:59.688433+0000","last_undegraded":"2023-09-23T08:24:59.688433+0000","last_fullsized":"2023-09-23T08:24:59.688433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"98'276945","reported_seq":554009,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:11.963885+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:25:11.963885+0000","last_peered":"2023-09-23T08:25:11.963885+0000","last_clean":"2023-09-23T08:25:11.963885+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:25:11.963885+0000","last_undegraded":"2023-09-23T08:25:11.963885+0000","last_fullsized":"2023-09-23T08:25:11.963885+0000","mapping_epoch":18,"log_start":"95'266911","ondisk_log_start":"95'266911","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2136,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 379s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5238587392,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":278017,"num_read_kb":1090608,"num_write":275588,"num_write_kb":1102352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'4777","reported_seq":7323,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:11.314461+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:25:11.314461+0000","last_peered":"2023-09-23T08:25:11.314461+0000","last_clean":"2023-09-23T08:25:11.314461+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:25:11.314461+0000","last_undegraded":"2023-09-23T08:25:11.314461+0000","last_fullsized":"2023-09-23T08:25:11.314461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"98'4783","reported_seq":9137,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:09.931875+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:25:09.931875+0000","last_peered":"2023-09-23T08:25:09.931875+0000","last_clean":"2023-09-23T08:25:09.931875+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:25:09.931875+0000","last_undegraded":"2023-09-23T08:25:09.931875+0000","last_fullsized":"2023-09-23T08:25:09.931875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20218945555,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1073428,"num_read_kb":4209436,"num_write":1061545,"num_write_kb":4246174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12891222016,"data_stored":54487990272,"data_compressed":36582966,"data_compressed_allocated":1685221376,"data_compressed_original":41891614720,"omap_allocated":0,"internal_metadata":0},"log_size":40193,"ondisk_log_size":40193,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37021,"ondisk_log_size":37021,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5458344,"kb_used_data":4196192,"kb_used_omap":573,"kb_used_meta":1261570,"kb_avail":88913496,"statfs":{"total":96636764160,"available":91047419904,"internally_reserved":0,"allocated":4296900608,"data_stored":18163038149,"data_compressed":11790023,"data_compressed_allocated":561737728,"data_compressed_original":13965115392,"omap_allocated":587472,"internal_metadata":1291847984},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":1.976,"15min":2.0870000000000002},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":14.285,"5min":14.285,"15min":14.285},"last":5.3390000000000004},{"interface":"front","average":{"1min":2.8780000000000001,"5min":1.9430000000000001,"15min":2.246},"min":{"1min":0.215,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":14.321,"5min":14.321,"15min":22.266999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":2.8639999999999999,"15min":3.097},"min":{"1min":0.26100000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.1839999999999993,"5min":16.27,"15min":23.045999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":2.5470000000000002,"5min":2.1989999999999998,"15min":2.7349999999999999},"min":{"1min":0.29499999999999998,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.081,"5min":12.044,"15min":28.257000000000001},"last":0.313}]}]},{"osd":0,"up_from":8,"seq":34359739243,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5586104,"kb_used_data":4197872,"kb_used_omap":580,"kb_used_meta":1387643,"kb_avail":88785736,"statfs":{"total":96636764160,"available":90916593664,"internally_reserved":0,"allocated":4298620928,"data_stored":18163702546,"data_compressed":12389680,"data_compressed_allocated":561778688,"data_compressed_original":13964251136,"omap_allocated":594767,"internal_metadata":1420946609},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,5,3,7,5,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.415,"15min":2.2469999999999999},"min":{"1min":0.191,"5min":0.182,"15min":0.17199999999999999},"max":{"1min":7.3579999999999997,"5min":12.529999999999999,"15min":19.885999999999999},"last":0.252},{"interface":"front","average":{"1min":3.1749999999999998,"5min":2.9079999999999999,"15min":3.069},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.507999999999999,"5min":15.093,"15min":24.385000000000002},"last":6.266}]},{"osd":2,"last update":"Sat Sep 23 08:24:16 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":3.0059999999999998,"15min":3.351},"min":{"1min":0.29999999999999999,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":7.1440000000000001,"5min":15.138,"15min":25.84},"last":6.3700000000000001},{"interface":"front","average":{"1min":2.2909999999999999,"5min":2.4060000000000001,"15min":2.7370000000000001},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.4100000000000001,"5min":12.609,"15min":25.120999999999999},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359739242,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5604920,"kb_used_data":4197936,"kb_used_omap":583,"kb_used_meta":1406392,"kb_avail":88766920,"statfs":{"total":96636764160,"available":90897326080,"internally_reserved":0,"allocated":4298686464,"data_stored":18163694736,"data_compressed":12409627,"data_compressed_allocated":561778688,"data_compressed_original":13964410880,"omap_allocated":597990,"internal_metadata":1440145434},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":4.0069999999999997,"5min":2.6960000000000002,"15min":2.7170000000000001},"min":{"1min":0.35399999999999998,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":29.236999999999998,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.379},{"interface":"front","average":{"1min":5.1539999999999999,"5min":3.242,"15min":3.335},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":29.521000000000001,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:24:27 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.7789999999999999,"15min":3.419},"min":{"1min":0.251,"5min":0.251,"15min":0.19},"max":{"1min":16.527000000000001,"5min":18.943999999999999,"15min":20.216999999999999},"last":10.58},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.601,"15min":1.7769999999999999},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.193},"max":{"1min":10.224,"5min":11.871,"15min":18.353000000000002},"last":6.181}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4297863168,"data_stored":18162700288,"data_compressed":12386698,"data_compressed_allocated":561741824,"data_compressed_original":13963149312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4297437184,"data_stored":18162647040,"data_compressed":12406228,"data_compressed_allocated":561741824,"data_compressed_original":13963522048,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4295921664,"data_stored":18162642944,"data_compressed":11790040,"data_compressed_allocated":561737728,"data_compressed_original":13964943360,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:25:14.586 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:14.586 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:14.586 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:14.586 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:25:34.588 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:25:35.379 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:25:35.389 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:25:35.411 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2303,"stamp":"2023-09-23T08:25:35.217814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20233651804,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1157983,"num_read_kb":4293180,"num_write":1085227,"num_write_kb":4283073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77277,"ondisk_log_size":77277,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16922440,"kb_used_data":12630120,"kb_used_omap":1738,"kb_used_meta":4290549,"kb_avail":266193080,"statfs":{"total":289910292480,"available":272581713920,"internally_reserved":0,"allocated":12933242880,"data_stored":54490640231,"data_compressed":36587359,"data_compressed_allocated":1685266432,"data_compressed_original":41856737280,"omap_allocated":1780229,"internal_metadata":4393522683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,15,6,27,18,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6987776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2032,"num_read_kb":8080,"num_write":1990,"num_write_kb":7960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008049"},"pg_stats":[{"pgid":"2.7","version":"98'5625","reported_seq":9464,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.408522+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:25:30.408522+0000","last_peered":"2023-09-23T08:25:30.408522+0000","last_clean":"2023-09-23T08:25:30.408522+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:25:30.408522+0000","last_undegraded":"2023-09-23T08:25:30.408522+0000","last_fullsized":"2023-09-23T08:25:30.408522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'8206","reported_seq":14185,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.414348+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:25:30.414348+0000","last_peered":"2023-09-23T08:25:30.414348+0000","last_clean":"2023-09-23T08:25:30.414348+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:25:30.414348+0000","last_undegraded":"2023-09-23T08:25:30.414348+0000","last_fullsized":"2023-09-23T08:25:30.414348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'3995","reported_seq":6569,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.411583+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:25:30.411583+0000","last_peered":"2023-09-23T08:25:30.411583+0000","last_clean":"2023-09-23T08:25:30.411583+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:25:30.411583+0000","last_undegraded":"2023-09-23T08:25:30.411583+0000","last_fullsized":"2023-09-23T08:25:30.411583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"98'273774","reported_seq":548081,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:32.472363+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:25:32.472363+0000","last_peered":"2023-09-23T08:25:32.472363+0000","last_clean":"2023-09-23T08:25:32.472363+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:25:32.472363+0000","last_undegraded":"2023-09-23T08:25:32.472363+0000","last_fullsized":"2023-09-23T08:25:32.472363+0000","mapping_epoch":18,"log_start":"96'263730","ondisk_log_start":"96'263730","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2223,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 396s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5172412416,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":275270,"num_read_kb":1079136,"num_write":272392,"num_write_kb":1089568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3097,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.406467+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:25:30.406467+0000","last_peered":"2023-09-23T08:25:30.406467+0000","last_clean":"2023-09-23T08:25:30.406467+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:25:30.406467+0000","last_undegraded":"2023-09-23T08:25:30.406467+0000","last_fullsized":"2023-09-23T08:25:30.406467+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'258707","reported_seq":518251,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:31.602949+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:25:31.602949+0000","last_peered":"2023-09-23T08:25:31.602949+0000","last_clean":"2023-09-23T08:25:31.602949+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:25:31.602949+0000","last_undegraded":"2023-09-23T08:25:31.602949+0000","last_fullsized":"2023-09-23T08:25:31.602949+0000","mapping_epoch":18,"log_start":"95'248690","ondisk_log_start":"95'248690","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":641,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 70s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4901126163,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":260607,"num_read_kb":1022044,"num_write":257406,"num_write_kb":1029618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":270,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:24:35.729748+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:24:35.729748+0000","last_peered":"2023-09-23T08:24:35.729748+0000","last_clean":"2023-09-23T08:24:35.729748+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:24:35.729748+0000","last_undegraded":"2023-09-23T08:24:35.729748+0000","last_fullsized":"2023-09-23T08:24:35.729748+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'6782","reported_seq":10560,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.413825+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:25:30.413825+0000","last_peered":"2023-09-23T08:25:30.413825+0000","last_clean":"2023-09-23T08:25:30.413825+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:25:30.413825+0000","last_undegraded":"2023-09-23T08:25:30.413825+0000","last_fullsized":"2023-09-23T08:25:30.413825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'260397","reported_seq":522069,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:31.564556+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:25:31.564556+0000","last_peered":"2023-09-23T08:25:31.564556+0000","last_clean":"2023-09-23T08:25:31.564556+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:25:31.564556+0000","last_undegraded":"2023-09-23T08:25:31.564556+0000","last_fullsized":"2023-09-23T08:25:31.564556+0000","mapping_epoch":18,"log_start":"95'250396","ondisk_log_start":"95'250396","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4917329920,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":262566,"num_read_kb":1029700,"num_write":259105,"num_write_kb":1036420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3703,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:25.291919+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:25:25.291919+0000","last_peered":"2023-09-23T08:25:25.291919+0000","last_clean":"2023-09-23T08:25:25.291919+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:25:25.291919+0000","last_undegraded":"2023-09-23T08:25:25.291919+0000","last_fullsized":"2023-09-23T08:25:25.291919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"98'277983","reported_seq":556049,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:32.510984+0000","last_change":"2023-09-23T08:18:52.448980+0000","last_active":"2023-09-23T08:25:32.510984+0000","last_peered":"2023-09-23T08:25:32.510984+0000","last_clean":"2023-09-23T08:25:32.510984+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:25:32.510984+0000","last_undegraded":"2023-09-23T08:25:32.510984+0000","last_fullsized":"2023-09-23T08:25:32.510984+0000","mapping_epoch":18,"log_start":"96'267923","ondisk_log_start":"96'267923","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2269,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 400s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5242183680,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":279004,"num_read_kb":1094536,"num_write":276626,"num_write_kb":1106504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'4797","reported_seq":7352,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:32.204282+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:25:32.204282+0000","last_peered":"2023-09-23T08:25:32.204282+0000","last_clean":"2023-09-23T08:25:32.204282+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:25:32.204282+0000","last_undegraded":"2023-09-23T08:25:32.204282+0000","last_fullsized":"2023-09-23T08:25:32.204282+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4797,"log_dups_size":0,"ondisk_log_size":4797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"98'4786","reported_seq":9152,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:30.413702+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:25:30.413702+0000","last_peered":"2023-09-23T08:25:30.413702+0000","last_clean":"2023-09-23T08:25:30.413702+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:25:30.413702+0000","last_undegraded":"2023-09-23T08:25:30.413702+0000","last_fullsized":"2023-09-23T08:25:30.413702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20233052179,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1077447,"num_read_kb":4225416,"num_write":1065529,"num_write_kb":4262110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12930420736,"data_stored":54488297472,"data_compressed":36581849,"data_compressed_allocated":1685192704,"data_compressed_original":41854537728,"omap_allocated":0,"internal_metadata":0},"log_size":40122,"ondisk_log_size":40122,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37056,"ondisk_log_size":37056,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5558896,"kb_used_data":4210724,"kb_used_omap":573,"kb_used_meta":1347586,"kb_avail":88812944,"statfs":{"total":96636764160,"available":90944454656,"internally_reserved":0,"allocated":4311781376,"data_stored":18163169221,"data_compressed":11789540,"data_compressed_allocated":561729536,"data_compressed_original":13951057920,"omap_allocated":587472,"internal_metadata":1379928368},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,10,1,11,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":1.976,"15min":2.0870000000000002},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":14.285,"5min":14.285,"15min":14.285},"last":1.3320000000000001},{"interface":"front","average":{"1min":2.8780000000000001,"5min":1.9430000000000001,"15min":2.246},"min":{"1min":0.215,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":14.321,"5min":14.321,"15min":22.266999999999999},"last":0.85899999999999999}]},{"osd":1,"last update":"Sat Sep 23 08:24:39 2023","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":2.8639999999999999,"15min":3.097},"min":{"1min":0.26100000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":8.1839999999999993,"5min":16.27,"15min":23.045999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":2.5470000000000002,"5min":2.1989999999999998,"15min":2.7349999999999999},"min":{"1min":0.29499999999999998,"5min":0.23599999999999999,"15min":0.20599999999999999},"max":{"1min":11.081,"5min":12.044,"15min":28.257000000000001},"last":0.755}]}]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5697128,"kb_used_data":4209952,"kb_used_omap":580,"kb_used_meta":1486587,"kb_avail":88674712,"statfs":{"total":96636764160,"available":90802905088,"internally_reserved":0,"allocated":4310990848,"data_stored":18163743506,"data_compressed":12388886,"data_compressed_allocated":561766400,"data_compressed_original":13952495616,"omap_allocated":594767,"internal_metadata":1522265265},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,11,3,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9810000000000001,"5min":1.6379999999999999,"15min":2.1899999999999999},"min":{"1min":0.185,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":15.090999999999999,"5min":15.090999999999999,"15min":19.885999999999999},"last":2.2360000000000002},{"interface":"front","average":{"1min":3.4929999999999999,"5min":3.016,"15min":3.1819999999999999},"min":{"1min":0.23200000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.749000000000001,"5min":15.093,"15min":24.385000000000002},"last":2.746}]},{"osd":2,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":5.274,"5min":3.4300000000000002,"15min":3.456},"min":{"1min":0.26600000000000001,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":32.962000000000003,"5min":32.962000000000003,"15min":32.962000000000003},"last":1.206},{"interface":"front","average":{"1min":4.7530000000000001,"5min":3.0899999999999999,"15min":2.9180000000000001},"min":{"1min":0.38,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":32.088999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":12.101000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739246,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5666416,"kb_used_data":4209444,"kb_used_omap":583,"kb_used_meta":1456376,"kb_avail":88705424,"statfs":{"total":96636764160,"available":90834354176,"internally_reserved":0,"allocated":4310470656,"data_stored":18163727504,"data_compressed":12408933,"data_compressed_allocated":561770496,"data_compressed_original":13953183744,"omap_allocated":597990,"internal_metadata":1491329050},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,5,10,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":2.6989999999999998,"15min":2.6880000000000002},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":10.840999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":1.804},{"interface":"front","average":{"1min":2.907,"5min":3.1899999999999999,"15min":3.3180000000000001},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":11.538,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.97599999999999998}]},{"osd":2,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":3.6349999999999998,"5min":3.536,"15min":3.5259999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.19},"max":{"1min":11.106999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.498,"15min":1.726},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.193},"max":{"1min":6.181,"5min":11.871,"15min":18.353000000000002},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4309946368,"data_stored":18162765824,"data_compressed":12385873,"data_compressed_allocated":561729536,"data_compressed_original":13951696896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4309385216,"data_stored":18162769920,"data_compressed":12406456,"data_compressed_allocated":561733632,"data_compressed_original":13952245760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4311089152,"data_stored":18162761728,"data_compressed":11789520,"data_compressed_allocated":561729536,"data_compressed_original":13950595072,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:25:35.413 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:35.414 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:35.414 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:35.414 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:25:36.771 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad4814000 2023-09-23T08:25:36.772 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:25:36.772 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:25:36.772 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:25:36.772 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:25:36.772 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:25:36.773 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:25:36.774 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:25:36.774 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:25:36.774 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:25:36.768+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:25:53.086 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:25:53.080+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:25:53.086 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:25:53.080+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:25:55.418 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:25:56.157 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:25:56.197 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:25:56.215 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2314,"stamp":"2023-09-23T08:25:55.228786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20247557724,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1162195,"num_read_kb":4309956,"num_write":1089138,"num_write_kb":4298717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77427,"ondisk_log_size":77427,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17194584,"kb_used_data":12664436,"kb_used_omap":1738,"kb_used_meta":4528373,"kb_avail":265920936,"statfs":{"total":289910292480,"available":272303038464,"internally_reserved":0,"allocated":12968382464,"data_stored":54491197632,"data_compressed":36586261,"data_compressed_allocated":1685311488,"data_compressed_original":41823612928,"omap_allocated":1780229,"internal_metadata":4637054459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,8,14,7,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6979584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2198,"num_read_kb":8744,"num_write":1967,"num_write_kb":7868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007881"},"pg_stats":[{"pgid":"2.7","version":"99'5643","reported_seq":9495,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.925196+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:25:51.925196+0000","last_peered":"2023-09-23T08:25:51.925196+0000","last_clean":"2023-09-23T08:25:51.925196+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:25:51.925196+0000","last_undegraded":"2023-09-23T08:25:51.925196+0000","last_fullsized":"2023-09-23T08:25:51.925196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5643,"log_dups_size":0,"ondisk_log_size":5643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'8230","reported_seq":14226,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.927131+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:25:51.927131+0000","last_peered":"2023-09-23T08:25:51.927131+0000","last_clean":"2023-09-23T08:25:51.927131+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:25:51.927131+0000","last_undegraded":"2023-09-23T08:25:51.927131+0000","last_fullsized":"2023-09-23T08:25:51.927131+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'4007","reported_seq":6590,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.925471+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:25:51.925471+0000","last_peered":"2023-09-23T08:25:51.925471+0000","last_clean":"2023-09-23T08:25:51.925471+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:25:51.925471+0000","last_undegraded":"2023-09-23T08:25:51.925471+0000","last_fullsized":"2023-09-23T08:25:51.925471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"99'274765","reported_seq":550159,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:53.066774+0000","last_change":"2023-09-23T08:18:56.395244+0000","last_active":"2023-09-23T08:25:53.066774+0000","last_peered":"2023-09-23T08:25:53.066774+0000","last_clean":"2023-09-23T08:25:53.066774+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:25:53.066774+0000","last_undegraded":"2023-09-23T08:25:53.066774+0000","last_fullsized":"2023-09-23T08:25:53.066774+0000","mapping_epoch":18,"log_start":"96'264745","ondisk_log_start":"96'264745","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2352,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 416s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5175779328,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":276341,"num_read_kb":1083396,"num_write":273383,"num_write_kb":1093532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3106,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:50.898041+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:25:50.898041+0000","last_peered":"2023-09-23T08:25:50.898041+0000","last_clean":"2023-09-23T08:25:50.898041+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:25:50.898041+0000","last_undegraded":"2023-09-23T08:25:50.898041+0000","last_fullsized":"2023-09-23T08:25:50.898041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'259677","reported_seq":520311,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:25:51.939515+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:25:51.939515+0000","last_peered":"2023-09-23T08:25:51.939515+0000","last_clean":"2023-09-23T08:25:51.939515+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:25:51.939515+0000","last_undegraded":"2023-09-23T08:25:51.939515+0000","last_fullsized":"2023-09-23T08:25:51.939515+0000","mapping_epoch":18,"log_start":"96'249608","ondisk_log_start":"96'249608","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":775,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 91s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4904644627,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":261682,"num_read_kb":1026324,"num_write":258376,"num_write_kb":1033498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":271,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:01.083854+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:25:01.083854+0000","last_peered":"2023-09-23T08:25:01.083854+0000","last_clean":"2023-09-23T08:25:01.083854+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:25:01.083854+0000","last_undegraded":"2023-09-23T08:25:01.083854+0000","last_fullsized":"2023-09-23T08:25:01.083854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'6800","reported_seq":10592,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.912126+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:25:51.912126+0000","last_peered":"2023-09-23T08:25:51.912126+0000","last_clean":"2023-09-23T08:25:51.912126+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:25:51.912126+0000","last_undegraded":"2023-09-23T08:25:51.912126+0000","last_fullsized":"2023-09-23T08:25:51.912126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"99'261356","reported_seq":523984,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.925557+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:25:51.925557+0000","last_peered":"2023-09-23T08:25:51.925557+0000","last_clean":"2023-09-23T08:25:51.925557+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:25:51.925557+0000","last_undegraded":"2023-09-23T08:25:51.925557+0000","last_fullsized":"2023-09-23T08:25:51.925557+0000","mapping_epoch":18,"log_start":"96'251306","ondisk_log_start":"96'251306","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4920819712,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":263522,"num_read_kb":1033508,"num_write":260064,"num_write_kb":1040256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3708,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:45.829577+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:25:45.829577+0000","last_peered":"2023-09-23T08:25:45.829577+0000","last_clean":"2023-09-23T08:25:45.829577+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:25:45.829577+0000","last_undegraded":"2023-09-23T08:25:45.829577+0000","last_fullsized":"2023-09-23T08:25:45.829577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"99'278974","reported_seq":558170,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:53.075211+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:25:53.075211+0000","last_peered":"2023-09-23T08:25:53.075211+0000","last_clean":"2023-09-23T08:25:53.075211+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:25:53.075211+0000","last_undegraded":"2023-09-23T08:25:53.075211+0000","last_fullsized":"2023-09-23T08:25:53.075211+0000","mapping_epoch":18,"log_start":"96'268941","ondisk_log_start":"96'268941","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5245714432,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":280114,"num_read_kb":1098964,"num_write":277617,"num_write_kb":1110468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'4807","reported_seq":7371,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:50.890442+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:25:50.890442+0000","last_peered":"2023-09-23T08:25:50.890442+0000","last_clean":"2023-09-23T08:25:50.890442+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:25:50.890442+0000","last_undegraded":"2023-09-23T08:25:50.890442+0000","last_fullsized":"2023-09-23T08:25:50.890442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"99'4804","reported_seq":9183,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:25:51.925473+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:25:51.925473+0000","last_peered":"2023-09-23T08:25:51.925473+0000","last_clean":"2023-09-23T08:25:51.925473+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:25:51.925473+0000","last_undegraded":"2023-09-23T08:25:51.925473+0000","last_fullsized":"2023-09-23T08:25:51.925473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20246958099,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1081659,"num_read_kb":4242192,"num_write":1069440,"num_write_kb":4277754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12965376000,"data_stored":54488813568,"data_compressed":36580420,"data_compressed_allocated":1685233664,"data_compressed_original":41821532160,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37156,"ondisk_log_size":37156,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739254,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5673408,"kb_used_data":4222512,"kb_used_omap":573,"kb_used_meta":1450306,"kb_avail":88698432,"statfs":{"total":96636764160,"available":90827194368,"internally_reserved":0,"allocated":4323852288,"data_stored":18163333176,"data_compressed":11789577,"data_compressed_allocated":561737728,"data_compressed_original":13939650560,"omap_allocated":587472,"internal_metadata":1485113648},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.9369999999999998,"5min":2.2629999999999999,"15min":2.0960000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":10.956,"5min":14.285,"15min":14.285},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.3210000000000002,"5min":2.2949999999999999,"15min":2.2210000000000001},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.901999999999999,"5min":14.321,"15min":22.266999999999999},"last":0.246}]},{"osd":1,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6139999999999999,"5min":2.6899999999999999,"15min":2.9860000000000002},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.9600000000000009,"5min":14.371,"15min":23.045999999999999},"last":10.695},{"interface":"front","average":{"1min":1.881,"5min":2.0990000000000002,"15min":2.573},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":9.6210000000000004,"5min":11.916,"15min":28.257000000000001},"last":1.333}]}]},{"osd":0,"up_from":8,"seq":34359739251,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5758828,"kb_used_data":4221216,"kb_used_omap":580,"kb_used_meta":1537019,"kb_avail":88613012,"statfs":{"total":96636764160,"available":90739724288,"internally_reserved":0,"allocated":4322525184,"data_stored":18163936133,"data_compressed":12388479,"data_compressed_allocated":561786880,"data_compressed_original":13941624832,"omap_allocated":594767,"internal_metadata":1573907633},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9810000000000001,"5min":1.6379999999999999,"15min":2.1899999999999999},"min":{"1min":0.185,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":15.090999999999999,"5min":15.090999999999999,"15min":19.885999999999999},"last":4.4199999999999999},{"interface":"front","average":{"1min":3.4929999999999999,"5min":3.016,"15min":3.1819999999999999},"min":{"1min":0.23200000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.749000000000001,"5min":15.093,"15min":24.385000000000002},"last":6.3129999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":5.274,"5min":3.4300000000000002,"15min":3.456},"min":{"1min":0.26600000000000001,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":32.962000000000003,"5min":32.962000000000003,"15min":32.962000000000003},"last":8.8130000000000006},{"interface":"front","average":{"1min":4.7530000000000001,"5min":3.0899999999999999,"15min":2.9180000000000001},"min":{"1min":0.38,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":32.088999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":4.4669999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739250,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5762348,"kb_used_data":4220708,"kb_used_omap":583,"kb_used_meta":1541048,"kb_avail":88609492,"statfs":{"total":96636764160,"available":90736119808,"internally_reserved":0,"allocated":4322004992,"data_stored":18163928323,"data_compressed":12408205,"data_compressed_allocated":561786880,"data_compressed_original":13942337536,"omap_allocated":597990,"internal_metadata":1578033178},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,7,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":2.6989999999999998,"15min":2.6880000000000002},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":10.840999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":4.9349999999999996},{"interface":"front","average":{"1min":2.907,"5min":3.1899999999999999,"15min":3.3180000000000001},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":11.538,"5min":29.521000000000001,"15min":29.521000000000001},"last":6.5350000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":3.6349999999999998,"5min":3.536,"15min":3.5259999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.19},"max":{"1min":11.106999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":10.522},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.498,"15min":1.726},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.193},"max":{"1min":6.181,"5min":11.871,"15min":18.353000000000002},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4321656832,"data_stored":18162929664,"data_compressed":12385577,"data_compressed_allocated":561750016,"data_compressed_original":13940637696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4321021952,"data_stored":18162950144,"data_compressed":12405230,"data_compressed_allocated":561745920,"data_compressed_original":13941252096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4322697216,"data_stored":18162933760,"data_compressed":11789613,"data_compressed_allocated":561737728,"data_compressed_original":13939642368,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:25:56.216 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:56.216 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:25:56.217 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:26:16.218 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:26:16.905 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:26:16.935 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:26:16.951 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2324,"stamp":"2023-09-23T08:26:15.241050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20261996124,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1166603,"num_read_kb":4327496,"num_write":1093262,"num_write_kb":4315213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77710,"ondisk_log_size":77710,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17453584,"kb_used_data":12700832,"kb_used_omap":1738,"kb_used_meta":4750965,"kb_avail":265661936,"statfs":{"total":289910292480,"available":272037822464,"internally_reserved":0,"allocated":13005651968,"data_stored":54491558080,"data_compressed":36582560,"data_compressed_allocated":1685172224,"data_compressed_original":41788170240,"omap_allocated":1780229,"internal_metadata":4864988667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,10,5,8,9,9,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9175040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2810,"num_read_kb":11176,"num_write":2628,"num_write_kb":10512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004675"},"pg_stats":[{"pgid":"2.7","version":"99'5682","reported_seq":9546,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.368724+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:26:11.368724+0000","last_peered":"2023-09-23T08:26:11.368724+0000","last_clean":"2023-09-23T08:26:11.368724+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:26:11.368724+0000","last_undegraded":"2023-09-23T08:26:11.368724+0000","last_fullsized":"2023-09-23T08:26:11.368724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'8282","reported_seq":14290,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.368360+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:26:11.368360+0000","last_peered":"2023-09-23T08:26:11.368360+0000","last_clean":"2023-09-23T08:26:11.368360+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:26:11.368360+0000","last_undegraded":"2023-09-23T08:26:11.368360+0000","last_fullsized":"2023-09-23T08:26:11.368360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'4033","reported_seq":6623,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.370067+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:26:11.370067+0000","last_peered":"2023-09-23T08:26:11.370067+0000","last_clean":"2023-09-23T08:26:11.370067+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:26:11.370067+0000","last_undegraded":"2023-09-23T08:26:11.370067+0000","last_fullsized":"2023-09-23T08:26:11.370067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4033,"log_dups_size":0,"ondisk_log_size":4033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"99'275801","reported_seq":552379,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:13.483869+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:26:13.483869+0000","last_peered":"2023-09-23T08:26:13.483869+0000","last_clean":"2023-09-23T08:26:13.483869+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:26:13.483869+0000","last_undegraded":"2023-09-23T08:26:13.483869+0000","last_fullsized":"2023-09-23T08:26:13.483869+0000","mapping_epoch":18,"log_start":"96'265762","ondisk_log_start":"96'265762","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5179432960,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":277518,"num_read_kb":1088084,"num_write":274419,"num_write_kb":1097676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3114,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.370180+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:26:11.370180+0000","last_peered":"2023-09-23T08:26:11.370180+0000","last_clean":"2023-09-23T08:26:11.370180+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:26:11.370180+0000","last_undegraded":"2023-09-23T08:26:11.370180+0000","last_fullsized":"2023-09-23T08:26:11.370180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'260705","reported_seq":522414,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:26:12.257069+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:26:12.257069+0000","last_peered":"2023-09-23T08:26:12.257069+0000","last_clean":"2023-09-23T08:26:12.257069+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:26:12.257069+0000","last_undegraded":"2023-09-23T08:26:12.257069+0000","last_fullsized":"2023-09-23T08:26:12.257069+0000","mapping_epoch":18,"log_start":"96'250626","ondisk_log_start":"96'250626","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":895,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 111s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4908310547,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":262743,"num_read_kb":1030544,"num_write":259404,"num_write_kb":1037610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":271,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:01.083854+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:25:01.083854+0000","last_peered":"2023-09-23T08:25:01.083854+0000","last_clean":"2023-09-23T08:25:01.083854+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:25:01.083854+0000","last_undegraded":"2023-09-23T08:25:01.083854+0000","last_fullsized":"2023-09-23T08:25:01.083854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'6839","reported_seq":10640,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.369425+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:26:11.369425+0000","last_peered":"2023-09-23T08:26:11.369425+0000","last_clean":"2023-09-23T08:26:11.369425+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:26:11.369425+0000","last_undegraded":"2023-09-23T08:26:11.369425+0000","last_fullsized":"2023-09-23T08:26:11.369425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"99'262362","reported_seq":526012,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:12.217762+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:26:12.217762+0000","last_peered":"2023-09-23T08:26:12.217762+0000","last_clean":"2023-09-23T08:26:12.217762+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:26:12.217762+0000","last_undegraded":"2023-09-23T08:26:12.217762+0000","last_fullsized":"2023-09-23T08:26:12.217762+0000","mapping_epoch":18,"log_start":"96'252318","ondisk_log_start":"96'252318","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4924321792,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":264542,"num_read_kb":1037556,"num_write":261070,"num_write_kb":1044280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3712,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:06.249649+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:26:06.249649+0000","last_peered":"2023-09-23T08:26:06.249649+0000","last_clean":"2023-09-23T08:26:06.249649+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:26:06.249649+0000","last_undegraded":"2023-09-23T08:26:06.249649+0000","last_fullsized":"2023-09-23T08:26:06.249649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"99'280028","reported_seq":560374,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:13.478953+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:26:13.478953+0000","last_peered":"2023-09-23T08:26:13.478953+0000","last_clean":"2023-09-23T08:26:13.478953+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:26:13.478953+0000","last_undegraded":"2023-09-23T08:26:13.478953+0000","last_fullsized":"2023-09-23T08:26:13.478953+0000","mapping_epoch":18,"log_start":"96'269952","ondisk_log_start":"96'269952","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5249331200,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":281264,"num_read_kb":1103548,"num_write":278671,"num_write_kb":1114684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'4829","reported_seq":7401,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:12.402395+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:26:12.402395+0000","last_peered":"2023-09-23T08:26:12.402395+0000","last_clean":"2023-09-23T08:26:12.402395+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:26:12.402395+0000","last_undegraded":"2023-09-23T08:26:12.402395+0000","last_fullsized":"2023-09-23T08:26:12.402395+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4829,"log_dups_size":0,"ondisk_log_size":4829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"99'4843","reported_seq":9232,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:11.372679+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:26:11.372679+0000","last_peered":"2023-09-23T08:26:11.372679+0000","last_clean":"2023-09-23T08:26:11.372679+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:26:11.372679+0000","last_undegraded":"2023-09-23T08:26:11.372679+0000","last_fullsized":"2023-09-23T08:26:11.372679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20261396499,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1086067,"num_read_kb":4259732,"num_write":1073564,"num_write_kb":4294250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13002936320,"data_stored":54489255936,"data_compressed":36577390,"data_compressed_allocated":1685102592,"data_compressed_original":41785876480,"omap_allocated":0,"internal_metadata":0},"log_size":40238,"ondisk_log_size":40238,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37373,"ondisk_log_size":37373,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5735032,"kb_used_data":4234148,"kb_used_omap":573,"kb_used_meta":1500290,"kb_avail":88636808,"statfs":{"total":96636764160,"available":90764091392,"internally_reserved":0,"allocated":4335767552,"data_stored":18163456056,"data_compressed":11789284,"data_compressed_allocated":561700864,"data_compressed_original":13928312832,"omap_allocated":587472,"internal_metadata":1536297264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.9369999999999998,"5min":2.2629999999999999,"15min":2.0960000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":10.956,"5min":14.285,"15min":14.285},"last":7.0190000000000001},{"interface":"front","average":{"1min":3.3210000000000002,"5min":2.2949999999999999,"15min":2.2210000000000001},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.901999999999999,"5min":14.321,"15min":22.266999999999999},"last":1.2390000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6139999999999999,"5min":2.6899999999999999,"15min":2.9860000000000002},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.9600000000000009,"5min":14.371,"15min":23.045999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.0990000000000002,"15min":2.573},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":9.6210000000000004,"5min":11.916,"15min":28.257000000000001},"last":0.71099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739255,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5857516,"kb_used_data":4233828,"kb_used_omap":580,"kb_used_meta":1623099,"kb_avail":88514324,"statfs":{"total":96636764160,"available":90638667776,"internally_reserved":0,"allocated":4335439872,"data_stored":18164063109,"data_compressed":12387076,"data_compressed_allocated":561737728,"data_compressed_original":13929353216,"omap_allocated":594767,"internal_metadata":1662053553},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,8,8,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9810000000000001,"5min":1.6379999999999999,"15min":2.1899999999999999},"min":{"1min":0.185,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":15.090999999999999,"5min":15.090999999999999,"15min":19.885999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":3.4929999999999999,"5min":3.016,"15min":3.1819999999999999},"min":{"1min":0.23200000000000001,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":11.749000000000001,"5min":15.093,"15min":24.385000000000002},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Sep 23 08:25:16 2023","interfaces":[{"interface":"back","average":{"1min":5.274,"5min":3.4300000000000002,"15min":3.456},"min":{"1min":0.26600000000000001,"5min":0.223,"15min":0.19500000000000001},"max":{"1min":32.962000000000003,"5min":32.962000000000003,"15min":32.962000000000003},"last":1.9990000000000001},{"interface":"front","average":{"1min":4.7530000000000001,"5min":3.0899999999999999,"15min":2.9180000000000001},"min":{"1min":0.38,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":32.088999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":1.954}]}]},{"osd":1,"up_from":8,"seq":34359739254,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5861036,"kb_used_data":4232856,"kb_used_omap":583,"kb_used_meta":1627576,"kb_avail":88510804,"statfs":{"total":96636764160,"available":90635063296,"internally_reserved":0,"allocated":4334444544,"data_stored":18164038915,"data_compressed":12406200,"data_compressed_allocated":561733632,"data_compressed_original":13930504192,"omap_allocated":597990,"internal_metadata":1666637850},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,0,1,4,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":2.6989999999999998,"15min":2.6880000000000002},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.161},"max":{"1min":10.840999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.245},{"interface":"front","average":{"1min":2.907,"5min":3.1899999999999999,"15min":3.3180000000000001},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":11.538,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.91100000000000003}]},{"osd":2,"last update":"Sat Sep 23 08:25:29 2023","interfaces":[{"interface":"back","average":{"1min":3.6349999999999998,"5min":3.536,"15min":3.5259999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.19},"max":{"1min":11.106999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.498,"15min":1.726},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.193},"max":{"1min":6.181,"5min":11.871,"15min":18.353000000000002},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4334186496,"data_stored":18163081216,"data_compressed":12384435,"data_compressed_allocated":561700864,"data_compressed_original":13928747008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4333469696,"data_stored":18163089408,"data_compressed":12403453,"data_compressed_allocated":561700864,"data_compressed_original":13929447424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4335280128,"data_stored":18163085312,"data_compressed":11789502,"data_compressed_allocated":561700864,"data_compressed_original":13927682048,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:26:16.953 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:26:16.954 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:26:36.955 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:26:37.614 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:26:37.627 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:26:37.651 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2335,"stamp":"2023-09-23T08:26:37.259468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20276975196,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1171045,"num_read_kb":4345192,"num_write":1097541,"num_write_kb":4332329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77873,"ondisk_log_size":77873,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17661852,"kb_used_data":12740864,"kb_used_omap":1738,"kb_used_meta":4919221,"kb_avail":265453668,"statfs":{"total":289910292480,"available":271824556032,"internally_reserved":0,"allocated":13046644736,"data_stored":54491992256,"data_compressed":36584244,"data_compressed_allocated":1685200896,"data_compressed_original":41749504000,"omap_allocated":1780229,"internal_metadata":5037282811},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,8,5,18,21,15,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7393280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2302,"num_read_kb":9176,"num_write":2125,"num_write_kb":8500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012217"},"pg_stats":[{"pgid":"2.7","version":"99'5706","reported_seq":9582,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.424087+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:26:32.424087+0000","last_peered":"2023-09-23T08:26:32.424087+0000","last_clean":"2023-09-23T08:26:32.424087+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:26:32.424087+0000","last_undegraded":"2023-09-23T08:26:32.424087+0000","last_fullsized":"2023-09-23T08:26:32.424087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'8314","reported_seq":14339,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.432695+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:26:32.432695+0000","last_peered":"2023-09-23T08:26:32.432695+0000","last_clean":"2023-09-23T08:26:32.432695+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:26:32.432695+0000","last_undegraded":"2023-09-23T08:26:32.432695+0000","last_fullsized":"2023-09-23T08:26:32.432695+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'4049","reported_seq":6647,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.423158+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:26:32.423158+0000","last_peered":"2023-09-23T08:26:32.423158+0000","last_clean":"2023-09-23T08:26:32.423158+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:26:32.423158+0000","last_undegraded":"2023-09-23T08:26:32.423158+0000","last_fullsized":"2023-09-23T08:26:32.423158+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4049,"log_dups_size":0,"ondisk_log_size":4049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"99'276852","reported_seq":554584,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:33.856334+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:26:33.856334+0000","last_peered":"2023-09-23T08:26:33.856334+0000","last_clean":"2023-09-23T08:26:33.856334+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:26:33.856334+0000","last_undegraded":"2023-09-23T08:26:33.856334+0000","last_fullsized":"2023-09-23T08:26:33.856334+0000","mapping_epoch":18,"log_start":"96'266777","ondisk_log_start":"96'266777","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5183152128,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":278675,"num_read_kb":1092696,"num_write":275470,"num_write_kb":1101880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3122,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:31.852149+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:26:31.852149+0000","last_peered":"2023-09-23T08:26:31.852149+0000","last_clean":"2023-09-23T08:26:31.852149+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:26:31.852149+0000","last_undegraded":"2023-09-23T08:26:31.852149+0000","last_fullsized":"2023-09-23T08:26:31.852149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'261799","reported_seq":524589,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:26:32.774342+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:26:32.774342+0000","last_peered":"2023-09-23T08:26:32.774342+0000","last_clean":"2023-09-23T08:26:32.774342+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:26:32.774342+0000","last_undegraded":"2023-09-23T08:26:32.774342+0000","last_fullsized":"2023-09-23T08:26:32.774342+0000","mapping_epoch":18,"log_start":"96'251744","ondisk_log_start":"96'251744","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1002,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 131s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4912140307,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":263813,"num_read_kb":1034808,"num_write":260498,"num_write_kb":1041986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":271,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:01.083854+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:25:01.083854+0000","last_peered":"2023-09-23T08:25:01.083854+0000","last_clean":"2023-09-23T08:25:01.083854+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:25:01.083854+0000","last_undegraded":"2023-09-23T08:25:01.083854+0000","last_fullsized":"2023-09-23T08:25:01.083854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'6863","reported_seq":10676,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.471070+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:26:32.471070+0000","last_peered":"2023-09-23T08:26:32.471070+0000","last_clean":"2023-09-23T08:26:32.471070+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:26:32.471070+0000","last_undegraded":"2023-09-23T08:26:32.471070+0000","last_fullsized":"2023-09-23T08:26:32.471070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"99'263395","reported_seq":528072,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.762090+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:26:32.762090+0000","last_peered":"2023-09-23T08:26:32.762090+0000","last_clean":"2023-09-23T08:26:32.762090+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:26:32.762090+0000","last_undegraded":"2023-09-23T08:26:32.762090+0000","last_fullsized":"2023-09-23T08:26:32.762090+0000","mapping_epoch":18,"log_start":"96'253332","ondisk_log_start":"96'253332","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4927913984,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":265571,"num_read_kb":1041648,"num_write":262103,"num_write_kb":1048412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3716,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:26.729714+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:26:26.729714+0000","last_peered":"2023-09-23T08:26:26.729714+0000","last_clean":"2023-09-23T08:26:26.729714+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:26:26.729714+0000","last_undegraded":"2023-09-23T08:26:26.729714+0000","last_fullsized":"2023-09-23T08:26:26.729714+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"99'281129","reported_seq":562659,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:33.899434+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:26:33.899434+0000","last_peered":"2023-09-23T08:26:33.899434+0000","last_clean":"2023-09-23T08:26:33.899434+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:26:33.899434+0000","last_undegraded":"2023-09-23T08:26:33.899434+0000","last_fullsized":"2023-09-23T08:26:33.899434+0000","mapping_epoch":18,"log_start":"96'271061","ondisk_log_start":"96'271061","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5253169152,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":282450,"num_read_kb":1108276,"num_write":279772,"num_write_kb":1119088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'4849","reported_seq":7429,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:33.132444+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:26:33.132444+0000","last_peered":"2023-09-23T08:26:33.132444+0000","last_clean":"2023-09-23T08:26:33.132444+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:26:33.132444+0000","last_undegraded":"2023-09-23T08:26:33.132444+0000","last_fullsized":"2023-09-23T08:26:33.132444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"99'4867","reported_seq":9268,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:32.426913+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:26:32.426913+0000","last_peered":"2023-09-23T08:26:32.426913+0000","last_clean":"2023-09-23T08:26:32.426913+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:26:32.426913+0000","last_undegraded":"2023-09-23T08:26:32.426913+0000","last_fullsized":"2023-09-23T08:26:32.426913+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20276375571,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1090509,"num_read_kb":4277428,"num_write":1077843,"num_write_kb":4311366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13042782208,"data_stored":54489628672,"data_compressed":36578615,"data_compressed_allocated":1685127168,"data_compressed_original":41748250624,"omap_allocated":0,"internal_metadata":0},"log_size":40261,"ondisk_log_size":40261,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37513,"ondisk_log_size":37513,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739262,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5816848,"kb_used_data":4247556,"kb_used_omap":573,"kb_used_meta":1568706,"kb_avail":88554992,"statfs":{"total":96636764160,"available":90680311808,"internally_reserved":0,"allocated":4349497344,"data_stored":18163603512,"data_compressed":11790444,"data_compressed_allocated":561709056,"data_compressed_original":13915361280,"omap_allocated":587472,"internal_metadata":1606355248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.9369999999999998,"5min":2.2629999999999999,"15min":2.0960000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":10.956,"5min":14.285,"15min":14.285},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.3210000000000002,"5min":2.2949999999999999,"15min":2.2210000000000001},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.901999999999999,"5min":14.321,"15min":22.266999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Sep 23 08:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6139999999999999,"5min":2.6899999999999999,"15min":2.9860000000000002},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.9600000000000009,"5min":14.371,"15min":23.045999999999999},"last":2.6850000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.0990000000000002,"15min":2.573},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":9.6210000000000004,"5min":11.916,"15min":28.257000000000001},"last":2.6480000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739259,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920500,"kb_used_data":4246892,"kb_used_omap":580,"kb_used_meta":1673019,"kb_avail":88451340,"statfs":{"total":96636764160,"available":90574172160,"internally_reserved":0,"allocated":4348817408,"data_stored":18164185989,"data_compressed":12387417,"data_compressed_allocated":561745920,"data_compressed_original":13916712960,"omap_allocated":594767,"internal_metadata":1713171633},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,1,10,4,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":2.6259999999999999,"5min":1.8819999999999999,"15min":2.0819999999999999},"min":{"1min":0.216,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":8.9649999999999999,"5min":15.090999999999999,"15min":15.090999999999999},"last":4.7169999999999996},{"interface":"front","average":{"1min":3.8290000000000002,"5min":3.2109999999999999,"15min":3.145},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":15.265000000000001,"5min":15.265000000000001,"15min":15.525},"last":0.65900000000000003}]},{"osd":2,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":4.2489999999999997,"5min":3.5800000000000001,"15min":3.544},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19500000000000001},"max":{"1min":21.331,"5min":32.962000000000003,"15min":32.962000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":3.972,"5min":3.6309999999999998,"15min":2.9540000000000002},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":14.653,"5min":32.088999999999999,"15min":32.088999999999999},"last":0.246}]}]},{"osd":1,"up_from":8,"seq":34359739258,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5924504,"kb_used_data":4246416,"kb_used_omap":583,"kb_used_meta":1677496,"kb_avail":88447336,"statfs":{"total":96636764160,"available":90570072064,"internally_reserved":0,"allocated":4348329984,"data_stored":18164202755,"data_compressed":12406383,"data_compressed_allocated":561745920,"data_compressed_original":13917429760,"omap_allocated":597990,"internal_metadata":1717755930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,6,15,6,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":4.1459999999999999,"5min":3.0139999999999998,"15min":2.6859999999999999},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.161},"max":{"1min":11.69,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":5.0039999999999996,"5min":3.6259999999999999,"15min":3.3490000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":14.664999999999999,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.26400000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":5.5499999999999998,"5min":3.8879999999999999,"15min":3.6789999999999998},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.19},"max":{"1min":14.301,"5min":18.943999999999999,"15min":20.216999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.7410000000000001,"5min":1.7709999999999999,"15min":1.7509999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":8.4329999999999998,"5min":11.871,"15min":18.353000000000002},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4347580416,"data_stored":18163212288,"data_compressed":12384547,"data_compressed_allocated":561709056,"data_compressed_original":13916098560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4346818560,"data_stored":18163195904,"data_compressed":12403552,"data_compressed_allocated":561709056,"data_compressed_original":13916823552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4348383232,"data_stored":18163220480,"data_compressed":11790516,"data_compressed_allocated":561709056,"data_compressed_original":13915328512,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:26:37.653 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:26:37.653 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:26:49.335 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:26:49.332+0000 7fea0cf80700 -1 reset not still connected to 0x55e255978ea0 2023-09-23T08:26:49.336 INFO:tasks.ceph.osd.0.smithi140.stderr:2023-09-23T08:26:49.332+0000 7fea0cf80700 -1 reset not still connected to 0x55e2566d5d40 2023-09-23T08:26:57.654 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:26:58.551 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:26:58.566 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:26:58.592 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2346,"stamp":"2023-09-23T08:26:57.288300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20291651164,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1175663,"num_read_kb":4363488,"num_write":1101738,"num_write_kb":4349117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77937,"ondisk_log_size":77937,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17937028,"kb_used_data":12777260,"kb_used_omap":1738,"kb_used_meta":5158005,"kb_avail":265178492,"statfs":{"total":289910292480,"available":271542775808,"internally_reserved":0,"allocated":13083914240,"data_stored":54492098752,"data_compressed":36579347,"data_compressed_allocated":1685004288,"data_compressed_original":41713569792,"omap_allocated":1780229,"internal_metadata":5281797627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,17,7,26,15,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7471104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2230,"num_read_kb":8828,"num_write":2131,"num_write_kb":8524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.021002"},"pg_stats":[{"pgid":"2.7","version":"99'5730","reported_seq":9618,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.330533+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:26:52.330533+0000","last_peered":"2023-09-23T08:26:52.330533+0000","last_clean":"2023-09-23T08:26:52.330533+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:26:52.330533+0000","last_undegraded":"2023-09-23T08:26:52.330533+0000","last_fullsized":"2023-09-23T08:26:52.330533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'8346","reported_seq":14387,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.331220+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:26:52.331220+0000","last_peered":"2023-09-23T08:26:52.331220+0000","last_clean":"2023-09-23T08:26:52.331220+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:26:52.331220+0000","last_undegraded":"2023-09-23T08:26:52.331220+0000","last_fullsized":"2023-09-23T08:26:52.331220+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'4065","reported_seq":6671,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.330015+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:26:52.330015+0000","last_peered":"2023-09-23T08:26:52.330015+0000","last_clean":"2023-09-23T08:26:52.330015+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:26:52.330015+0000","last_undegraded":"2023-09-23T08:26:52.330015+0000","last_fullsized":"2023-09-23T08:26:52.330015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"99'277976","reported_seq":556916,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:54.408688+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:26:54.408688+0000","last_peered":"2023-09-23T08:26:54.408688+0000","last_clean":"2023-09-23T08:26:54.408688+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:26:54.408688+0000","last_undegraded":"2023-09-23T08:26:54.408688+0000","last_fullsized":"2023-09-23T08:26:54.408688+0000","mapping_epoch":18,"log_start":"97'267901","ondisk_log_start":"97'267901","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5187055616,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":279882,"num_read_kb":1097476,"num_write":276594,"num_write_kb":1106376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3131,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.343832+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:26:52.343832+0000","last_peered":"2023-09-23T08:26:52.343832+0000","last_clean":"2023-09-23T08:26:52.343832+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:26:52.343832+0000","last_undegraded":"2023-09-23T08:26:52.343832+0000","last_fullsized":"2023-09-23T08:26:52.343832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'262774","reported_seq":526685,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:26:53.194652+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:26:53.194652+0000","last_peered":"2023-09-23T08:26:53.194652+0000","last_clean":"2023-09-23T08:26:53.194652+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:26:53.194652+0000","last_undegraded":"2023-09-23T08:26:53.194652+0000","last_fullsized":"2023-09-23T08:26:53.194652+0000","mapping_epoch":18,"log_start":"96'252759","ondisk_log_start":"96'252759","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1177,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4915568659,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":264919,"num_read_kb":1039200,"num_write":261473,"num_write_kb":1045886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":271,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-09-23T08:25:01.083854+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:25:01.083854+0000","last_peered":"2023-09-23T08:25:01.083854+0000","last_clean":"2023-09-23T08:25:01.083854+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:25:01.083854+0000","last_undegraded":"2023-09-23T08:25:01.083854+0000","last_fullsized":"2023-09-23T08:25:01.083854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'6884","reported_seq":10709,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.348250+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:26:52.348250+0000","last_peered":"2023-09-23T08:26:52.348250+0000","last_clean":"2023-09-23T08:26:52.348250+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:26:52.348250+0000","last_undegraded":"2023-09-23T08:26:52.348250+0000","last_fullsized":"2023-09-23T08:26:52.348250+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6884,"log_dups_size":0,"ondisk_log_size":6884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"99'264392","reported_seq":530172,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:53.140034+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:26:53.140034+0000","last_peered":"2023-09-23T08:26:53.140034+0000","last_clean":"2023-09-23T08:26:53.140034+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:26:53.140034+0000","last_undegraded":"2023-09-23T08:26:53.140034+0000","last_fullsized":"2023-09-23T08:26:53.140034+0000","mapping_epoch":18,"log_start":"96'254342","ondisk_log_start":"96'254342","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4931407872,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":266671,"num_read_kb":1046004,"num_write":263100,"num_write_kb":1052400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3721,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.339711+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:26:52.339711+0000","last_peered":"2023-09-23T08:26:52.339711+0000","last_clean":"2023-09-23T08:26:52.339711+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:26:52.339711+0000","last_undegraded":"2023-09-23T08:26:52.339711+0000","last_fullsized":"2023-09-23T08:26:52.339711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"99'282230","reported_seq":564967,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:54.407144+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:26:54.407144+0000","last_peered":"2023-09-23T08:26:54.407144+0000","last_clean":"2023-09-23T08:26:54.407144+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:26:54.407144+0000","last_undegraded":"2023-09-23T08:26:54.407144+0000","last_fullsized":"2023-09-23T08:26:54.407144+0000","mapping_epoch":18,"log_start":"97'272178","ondisk_log_start":"97'272178","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5257019392,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":283655,"num_read_kb":1113044,"num_write":280873,"num_write_kb":1123492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'4865","reported_seq":7453,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.331630+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:26:52.331630+0000","last_peered":"2023-09-23T08:26:52.331630+0000","last_clean":"2023-09-23T08:26:52.331630+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:26:52.331630+0000","last_undegraded":"2023-09-23T08:26:52.331630+0000","last_fullsized":"2023-09-23T08:26:52.331630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"99'4891","reported_seq":9304,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-09-23T08:26:52.331769+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:26:52.331769+0000","last_peered":"2023-09-23T08:26:52.331769+0000","last_clean":"2023-09-23T08:26:52.331769+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:26:52.331769+0000","last_undegraded":"2023-09-23T08:26:52.331769+0000","last_fullsized":"2023-09-23T08:26:52.331769+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20291051539,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1095127,"num_read_kb":4295724,"num_write":1082040,"num_write_kb":4328154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13080793088,"data_stored":54489731072,"data_compressed":36573770,"data_compressed_allocated":1684922368,"data_compressed_original":41711599616,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37646,"ondisk_log_size":37646,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739266,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5878960,"kb_used_data":4259688,"kb_used_omap":573,"kb_used_meta":1618690,"kb_avail":88492880,"statfs":{"total":96636764160,"available":90616709120,"internally_reserved":0,"allocated":4361920512,"data_stored":18163669048,"data_compressed":11790036,"data_compressed_allocated":561639424,"data_compressed_original":13903421440,"omap_allocated":587472,"internal_metadata":1657538864},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":2.9969999999999999,"5min":2.5449999999999999,"15min":2.165},"min":{"1min":0.214,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.093,"5min":14.285,"15min":14.285},"last":1.0069999999999999},{"interface":"front","average":{"1min":2.0899999999999999,"5min":2.3889999999999998,"15min":2.2189999999999999},"min":{"1min":0.246,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":11.135999999999999,"5min":14.321,"15min":22.266999999999999},"last":8.4329999999999998}]},{"osd":1,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":2.8599999999999999,"15min":3.077},"min":{"1min":0.28199999999999997,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":10.695,"5min":14.371,"15min":23.045999999999999},"last":6.0220000000000002},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.1240000000000001,"15min":2.52},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":4.6959999999999997,"5min":11.916,"15min":28.257000000000001},"last":6.0499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6035184,"kb_used_data":4258856,"kb_used_omap":580,"kb_used_meta":1775739,"kb_avail":88336656,"statfs":{"total":96636764160,"available":90456735744,"internally_reserved":0,"allocated":4361068544,"data_stored":18164210565,"data_compressed":12384997,"data_compressed_allocated":561680384,"data_compressed_original":13904887808,"omap_allocated":594767,"internal_metadata":1818356913},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,8,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":2.6259999999999999,"5min":1.8819999999999999,"15min":2.0819999999999999},"min":{"1min":0.216,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":8.9649999999999999,"5min":15.090999999999999,"15min":15.090999999999999},"last":9.1069999999999993},{"interface":"front","average":{"1min":3.8290000000000002,"5min":3.2109999999999999,"15min":3.145},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":15.265000000000001,"5min":15.265000000000001,"15min":15.525},"last":9.2889999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":4.2489999999999997,"5min":3.5800000000000001,"15min":3.544},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19500000000000001},"max":{"1min":21.331,"5min":32.962000000000003,"15min":32.962000000000003},"last":9.0779999999999994},{"interface":"front","average":{"1min":3.972,"5min":3.6309999999999998,"15min":2.9540000000000002},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":14.653,"5min":32.088999999999999,"15min":32.088999999999999},"last":9.3520000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739262,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6022884,"kb_used_data":4258716,"kb_used_omap":583,"kb_used_meta":1763576,"kb_avail":88348956,"statfs":{"total":96636764160,"available":90469330944,"internally_reserved":0,"allocated":4360925184,"data_stored":18164219139,"data_compressed":12404314,"data_compressed_allocated":561684480,"data_compressed_original":13905260544,"omap_allocated":597990,"internal_metadata":1805901850},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,5,17,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":4.1459999999999999,"5min":3.0139999999999998,"15min":2.6859999999999999},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.161},"max":{"1min":11.69,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.374},{"interface":"front","average":{"1min":5.0039999999999996,"5min":3.6259999999999999,"15min":3.3490000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":14.664999999999999,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":5.5499999999999998,"5min":3.8879999999999999,"15min":3.6789999999999998},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.19},"max":{"1min":14.301,"5min":18.943999999999999,"15min":20.216999999999999},"last":2.335},{"interface":"front","average":{"1min":2.7410000000000001,"5min":1.7709999999999999,"15min":1.7509999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":8.4329999999999998,"5min":11.871,"15min":18.353000000000002},"last":1.861}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4360290304,"data_stored":18163269632,"data_compressed":12382274,"data_compressed_allocated":561639424,"data_compressed_original":13903863808,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4359565312,"data_stored":18163236864,"data_compressed":12401598,"data_compressed_allocated":561643520,"data_compressed_original":13904527360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4360937472,"data_stored":18163224576,"data_compressed":11789898,"data_compressed_allocated":561639424,"data_compressed_original":13903208448,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:26:58.594 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:26:58.595 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:27:18.595 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:27:19.378 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:27:19.431 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:27:19.445 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2357,"stamp":"2023-09-23T08:27:19.302663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20308194908,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1180817,"num_read_kb":4383992,"num_write":1106452,"num_write_kb":4367973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77980,"ondisk_log_size":77980,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18200776,"kb_used_data":12817636,"kb_used_omap":1738,"kb_used_meta":5381365,"kb_avail":264914744,"statfs":{"total":289910292480,"available":271272697856,"internally_reserved":0,"allocated":13125259264,"data_stored":54492324248,"data_compressed":36574948,"data_compressed_allocated":1684840448,"data_compressed_original":41674190848,"omap_allocated":1780229,"internal_metadata":5510518267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,15,9,26,23,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9129984,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2921,"num_read_kb":11628,"num_write":2616,"num_write_kb":10464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002249"},"pg_stats":[{"pgid":"2.7","version":"100'5745","reported_seq":9649,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:17.930884+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:27:17.930884+0000","last_peered":"2023-09-23T08:27:17.930884+0000","last_clean":"2023-09-23T08:27:17.930884+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:27:17.930884+0000","last_undegraded":"2023-09-23T08:27:17.930884+0000","last_fullsized":"2023-09-23T08:27:17.930884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'8366","reported_seq":14428,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:17.931632+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:27:17.931632+0000","last_peered":"2023-09-23T08:27:17.931632+0000","last_clean":"2023-09-23T08:27:17.931632+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:27:17.931632+0000","last_undegraded":"2023-09-23T08:27:17.931632+0000","last_fullsized":"2023-09-23T08:27:17.931632+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'4075","reported_seq":6692,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:17.930720+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:27:17.930720+0000","last_peered":"2023-09-23T08:27:17.930720+0000","last_clean":"2023-09-23T08:27:17.930720+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:27:17.930720+0000","last_undegraded":"2023-09-23T08:27:17.930720+0000","last_fullsized":"2023-09-23T08:27:17.930720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'279051","reported_seq":559164,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:14.839099+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:27:14.839099+0000","last_peered":"2023-09-23T08:27:14.839099+0000","last_clean":"2023-09-23T08:27:14.839099+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:27:14.839099+0000","last_undegraded":"2023-09-23T08:27:14.839099+0000","last_fullsized":"2023-09-23T08:27:14.839099+0000","mapping_epoch":18,"log_start":"97'269006","ondisk_log_start":"97'269006","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5190746112,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":281050,"num_read_kb":1102116,"num_write":277669,"num_write_kb":1110676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3140,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:12.807721+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:27:12.807721+0000","last_peered":"2023-09-23T08:27:12.807721+0000","last_clean":"2023-09-23T08:27:12.807721+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:27:12.807721+0000","last_undegraded":"2023-09-23T08:27:12.807721+0000","last_fullsized":"2023-09-23T08:27:12.807721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'264021","reported_seq":529325,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:27:18.592077+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:27:18.592077+0000","last_peered":"2023-09-23T08:27:18.592077+0000","last_clean":"2023-09-23T08:27:18.592077+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:27:18.592077+0000","last_undegraded":"2023-09-23T08:27:18.592077+0000","last_fullsized":"2023-09-23T08:27:18.592077+0000","mapping_epoch":18,"log_start":"97'253987","ondisk_log_start":"97'253987","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1370,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4919992339,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":266293,"num_read_kb":1044668,"num_write":262720,"num_write_kb":1050874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":273,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:26:55.840802+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:26:55.840802+0000","last_peered":"2023-09-23T08:26:55.840802+0000","last_clean":"2023-09-23T08:26:55.840802+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:26:55.840802+0000","last_undegraded":"2023-09-23T08:26:55.840802+0000","last_fullsized":"2023-09-23T08:26:55.840802+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'6899","reported_seq":10740,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:17.943357+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:27:17.943357+0000","last_peered":"2023-09-23T08:27:17.943357+0000","last_clean":"2023-09-23T08:27:17.943357+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:27:17.943357+0000","last_undegraded":"2023-09-23T08:27:17.943357+0000","last_fullsized":"2023-09-23T08:27:17.943357+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"100'265670","reported_seq":532873,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:18.552082+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:27:18.552082+0000","last_peered":"2023-09-23T08:27:18.552082+0000","last_clean":"2023-09-23T08:27:18.552082+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:27:18.552082+0000","last_undegraded":"2023-09-23T08:27:18.552082+0000","last_fullsized":"2023-09-23T08:27:18.552082+0000","mapping_epoch":18,"log_start":"97'255650","ondisk_log_start":"97'255650","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4935864320,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":268095,"num_read_kb":1051676,"num_write":264378,"num_write_kb":1057512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3726,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:07.732635+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:27:07.732635+0000","last_peered":"2023-09-23T08:27:07.732635+0000","last_clean":"2023-09-23T08:27:07.732635+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:27:07.732635+0000","last_undegraded":"2023-09-23T08:27:07.732635+0000","last_fullsized":"2023-09-23T08:27:07.732635+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"100'283344","reported_seq":567269,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:14.881074+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:27:14.881074+0000","last_peered":"2023-09-23T08:27:14.881074+0000","last_clean":"2023-09-23T08:27:14.881074+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:27:14.881074+0000","last_undegraded":"2023-09-23T08:27:14.881074+0000","last_fullsized":"2023-09-23T08:27:14.881074+0000","mapping_epoch":18,"log_start":"97'273295","ondisk_log_start":"97'273295","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5260992512,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":284843,"num_read_kb":1117768,"num_write":281987,"num_write_kb":1127948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"100'4877","reported_seq":7474,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:14.438529+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:27:14.438529+0000","last_peered":"2023-09-23T08:27:14.438529+0000","last_clean":"2023-09-23T08:27:14.438529+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:27:14.438529+0000","last_undegraded":"2023-09-23T08:27:14.438529+0000","last_fullsized":"2023-09-23T08:27:14.438529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"100'4906","reported_seq":9335,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:17.931896+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:27:17.931896+0000","last_peered":"2023-09-23T08:27:17.931896+0000","last_clean":"2023-09-23T08:27:17.931896+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:27:17.931896+0000","last_undegraded":"2023-09-23T08:27:17.931896+0000","last_fullsized":"2023-09-23T08:27:17.931896+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20307595283,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1100281,"num_read_kb":4316228,"num_write":1086754,"num_write_kb":4347010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13122002944,"data_stored":54489849856,"data_compressed":36568071,"data_compressed_allocated":1684762624,"data_compressed_original":41672253440,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37733,"ondisk_log_size":37733,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739270,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5975788,"kb_used_data":4271840,"kb_used_omap":573,"kb_used_meta":1703362,"kb_avail":88396052,"statfs":{"total":96636764160,"available":90517557248,"internally_reserved":0,"allocated":4374364160,"data_stored":18163705984,"data_compressed":11789475,"data_compressed_allocated":561586176,"data_compressed_original":13891538944,"omap_allocated":587472,"internal_metadata":1744242992},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":2.9969999999999999,"5min":2.5449999999999999,"15min":2.165},"min":{"1min":0.214,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.093,"5min":14.285,"15min":14.285},"last":0.314},{"interface":"front","average":{"1min":2.0899999999999999,"5min":2.3889999999999998,"15min":2.2189999999999999},"min":{"1min":0.246,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":11.135999999999999,"5min":14.321,"15min":22.266999999999999},"last":0.499}]},{"osd":1,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":2.8599999999999999,"15min":3.077},"min":{"1min":0.28199999999999997,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":10.695,"5min":14.371,"15min":23.045999999999999},"last":8.3900000000000006},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.1240000000000001,"15min":2.52},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":4.6959999999999997,"5min":11.916,"15min":28.257000000000001},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739267,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6097868,"kb_used_data":4271620,"kb_used_omap":580,"kb_used_meta":1825659,"kb_avail":88273972,"statfs":{"total":96636764160,"available":90392547328,"internally_reserved":0,"allocated":4374138880,"data_stored":18164308941,"data_compressed":12383268,"data_compressed_allocated":561627136,"data_compressed_original":13892464640,"omap_allocated":594767,"internal_metadata":1869474993},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,10,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":2.6259999999999999,"5min":1.8819999999999999,"15min":2.0819999999999999},"min":{"1min":0.216,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":8.9649999999999999,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":3.8290000000000002,"5min":3.2109999999999999,"15min":3.145},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":15.265000000000001,"5min":15.265000000000001,"15min":15.525},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:26:17 2023","interfaces":[{"interface":"back","average":{"1min":4.2489999999999997,"5min":3.5800000000000001,"15min":3.544},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.19500000000000001},"max":{"1min":21.331,"5min":32.962000000000003,"15min":32.962000000000003},"last":1.083},{"interface":"front","average":{"1min":3.972,"5min":3.6309999999999998,"15min":2.9540000000000002},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.20699999999999999},"max":{"1min":14.653,"5min":32.088999999999999,"15min":32.088999999999999},"last":4.6269999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739267,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6127120,"kb_used_data":4274176,"kb_used_omap":583,"kb_used_meta":1852344,"kb_avail":88244720,"statfs":{"total":96636764160,"available":90362593280,"internally_reserved":0,"allocated":4376756224,"data_stored":18164309323,"data_compressed":12402205,"data_compressed_allocated":561627136,"data_compressed_original":13890187264,"omap_allocated":597990,"internal_metadata":1896800282},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,4,12,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":4.1459999999999999,"5min":3.0139999999999998,"15min":2.6859999999999999},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.161},"max":{"1min":11.69,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":5.0039999999999996,"5min":3.6259999999999999,"15min":3.3490000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":14.664999999999999,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Sep 23 08:26:30 2023","interfaces":[{"interface":"back","average":{"1min":5.5499999999999998,"5min":3.8879999999999999,"15min":3.6789999999999998},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.19},"max":{"1min":14.301,"5min":18.943999999999999,"15min":20.216999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":2.7410000000000001,"5min":1.7709999999999999,"15min":1.7509999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":8.4329999999999998,"5min":11.871,"15min":18.353000000000002},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4373061632,"data_stored":18163294208,"data_compressed":12380146,"data_compressed_allocated":561586176,"data_compressed_original":13891657728,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4375482368,"data_stored":18163302400,"data_compressed":12398810,"data_compressed_allocated":561590272,"data_compressed_original":13889343488,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4373458944,"data_stored":18163253248,"data_compressed":11789115,"data_compressed_allocated":561586176,"data_compressed_original":13891252224,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:27:19.447 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:27:19.447 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:27:39.448 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:27:40.065 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.058+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60340 2023-09-23T08:27:40.065 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.058+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5b60b60 2023-09-23T08:27:40.065 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.058+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb8820 2023-09-23T08:27:40.065 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.058+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5bb9790 2023-09-23T08:27:40.070 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad5dab1e0 2023-09-23T08:27:40.070 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603e9c0 2023-09-23T08:27:40.070 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad603fee0 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c15f0 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad61c1c70 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad64bc000 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6ab7d40 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bad6db1e10 2023-09-23T08:27:40.071 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55bada2f5ba0 2023-09-23T08:27:40.072 INFO:tasks.ceph.osd.1.smithi140.stderr:2023-09-23T08:27:40.066+0000 7fa4ae7fa700 -1 reset not still connected to 0x55badadf7ad0 2023-09-23T08:27:40.122 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:27:40.134 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:27:40.146 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2367,"stamp":"2023-09-23T08:27:39.318340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20323096156,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1185088,"num_read_kb":4401024,"num_write":1110700,"num_write_kb":4384965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78194,"ondisk_log_size":78194,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18460672,"kb_used_data":12855456,"kb_used_omap":1738,"kb_used_meta":5603445,"kb_avail":264654848,"statfs":{"total":289910292480,"available":271006564352,"internally_reserved":0,"allocated":13163986944,"data_stored":54492393880,"data_compressed":36571538,"data_compressed_allocated":1684819968,"data_compressed_original":41637425152,"omap_allocated":1780229,"internal_metadata":5737928187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,1,10,10,10,11,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9236480,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2701,"num_read_kb":10768,"num_write":2646,"num_write_kb":10584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012313"},"pg_stats":[{"pgid":"2.7","version":"100'5766","reported_seq":9682,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:38.411060+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:27:38.411060+0000","last_peered":"2023-09-23T08:27:38.411060+0000","last_clean":"2023-09-23T08:27:38.411060+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:27:38.411060+0000","last_undegraded":"2023-09-23T08:27:38.411060+0000","last_fullsized":"2023-09-23T08:27:38.411060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'8394","reported_seq":14472,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:38.411020+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:27:38.411020+0000","last_peered":"2023-09-23T08:27:38.411020+0000","last_clean":"2023-09-23T08:27:38.411020+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:27:38.411020+0000","last_undegraded":"2023-09-23T08:27:38.411020+0000","last_fullsized":"2023-09-23T08:27:38.411020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'4089","reported_seq":6714,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:38.411274+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:27:38.411274+0000","last_peered":"2023-09-23T08:27:38.411274+0000","last_clean":"2023-09-23T08:27:38.411274+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:27:38.411274+0000","last_undegraded":"2023-09-23T08:27:38.411274+0000","last_fullsized":"2023-09-23T08:27:38.411274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'280141","reported_seq":561325,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:35.219428+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:27:35.219428+0000","last_peered":"2023-09-23T08:27:35.219428+0000","last_clean":"2023-09-23T08:27:35.219428+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:27:35.219428+0000","last_undegraded":"2023-09-23T08:27:35.219428+0000","last_fullsized":"2023-09-23T08:27:35.219428+0000","mapping_epoch":18,"log_start":"97'270122","ondisk_log_start":"97'270122","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5194567680,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":282124,"num_read_kb":1106400,"num_write":278759,"num_write_kb":1115036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3148,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:33.292182+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:27:33.292182+0000","last_peered":"2023-09-23T08:27:33.292182+0000","last_clean":"2023-09-23T08:27:33.292182+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:27:33.292182+0000","last_undegraded":"2023-09-23T08:27:33.292182+0000","last_fullsized":"2023-09-23T08:27:33.292182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'265000","reported_seq":531362,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:27:39.102015+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:27:39.102015+0000","last_peered":"2023-09-23T08:27:39.102015+0000","last_clean":"2023-09-23T08:27:39.102015+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:27:39.102015+0000","last_undegraded":"2023-09-23T08:27:39.102015+0000","last_fullsized":"2023-09-23T08:27:39.102015+0000","mapping_epoch":18,"log_start":"97'254901","ondisk_log_start":"97'254901","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1522,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4923449363,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":267338,"num_read_kb":1048828,"num_write":263699,"num_write_kb":1054790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":273,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:26:55.840802+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:26:55.840802+0000","last_peered":"2023-09-23T08:26:55.840802+0000","last_clean":"2023-09-23T08:26:55.840802+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:26:55.840802+0000","last_undegraded":"2023-09-23T08:26:55.840802+0000","last_fullsized":"2023-09-23T08:26:55.840802+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'6920","reported_seq":10773,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:38.435824+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:27:38.435824+0000","last_peered":"2023-09-23T08:27:38.435824+0000","last_clean":"2023-09-23T08:27:38.435824+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:27:38.435824+0000","last_undegraded":"2023-09-23T08:27:38.435824+0000","last_fullsized":"2023-09-23T08:27:38.435824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"100'266733","reported_seq":534965,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:39.066323+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:27:39.066323+0000","last_peered":"2023-09-23T08:27:39.066323+0000","last_clean":"2023-09-23T08:27:39.066323+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:27:39.066323+0000","last_undegraded":"2023-09-23T08:27:39.066323+0000","last_fullsized":"2023-09-23T08:27:39.066323+0000","mapping_epoch":18,"log_start":"97'256658","ondisk_log_start":"97'256658","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4939546624,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":269123,"num_read_kb":1055776,"num_write":265441,"num_write_kb":1061764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3730,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:28.173972+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:27:28.173972+0000","last_peered":"2023-09-23T08:27:28.173972+0000","last_clean":"2023-09-23T08:27:28.173972+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:27:28.173972+0000","last_undegraded":"2023-09-23T08:27:28.173972+0000","last_fullsized":"2023-09-23T08:27:28.173972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"100'284460","reported_seq":569498,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:35.225096+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:27:35.225096+0000","last_peered":"2023-09-23T08:27:35.225096+0000","last_clean":"2023-09-23T08:27:35.225096+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:27:35.225096+0000","last_undegraded":"2023-09-23T08:27:35.225096+0000","last_fullsized":"2023-09-23T08:27:35.225096+0000","mapping_epoch":18,"log_start":"97'274408","ondisk_log_start":"97'274408","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5264932864,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":285967,"num_read_kb":1122256,"num_write":283103,"num_write_kb":1132412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"100'4889","reported_seq":7495,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:35.134153+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:27:35.134153+0000","last_peered":"2023-09-23T08:27:35.134153+0000","last_clean":"2023-09-23T08:27:35.134153+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:27:35.134153+0000","last_undegraded":"2023-09-23T08:27:35.134153+0000","last_fullsized":"2023-09-23T08:27:35.134153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"100'4927","reported_seq":9368,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:38.410870+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:27:38.410870+0000","last_peered":"2023-09-23T08:27:38.410870+0000","last_clean":"2023-09-23T08:27:38.410870+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:27:38.410870+0000","last_undegraded":"2023-09-23T08:27:38.410870+0000","last_fullsized":"2023-09-23T08:27:38.410870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20322496531,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1104552,"num_read_kb":4333260,"num_write":1091002,"num_write_kb":4364002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13161254912,"data_stored":54490038272,"data_compressed":36565299,"data_compressed_allocated":1684746240,"data_compressed_original":41635069952,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37850,"ondisk_log_size":37850,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739274,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6056428,"kb_used_data":4284516,"kb_used_omap":573,"kb_used_meta":1771330,"kb_avail":88315412,"statfs":{"total":96636764160,"available":90434981888,"internally_reserved":0,"allocated":4387344384,"data_stored":18163767424,"data_compressed":11789458,"data_compressed_allocated":561582080,"data_compressed_original":13879242752,"omap_allocated":587472,"internal_metadata":1813842224},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":2.9969999999999999,"5min":2.5449999999999999,"15min":2.165},"min":{"1min":0.214,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.093,"5min":14.285,"15min":14.285},"last":3.4470000000000001},{"interface":"front","average":{"1min":2.0899999999999999,"5min":2.3889999999999998,"15min":2.2189999999999999},"min":{"1min":0.246,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":11.135999999999999,"5min":14.321,"15min":22.266999999999999},"last":3.4860000000000002}]},{"osd":1,"last update":"Sat Sep 23 08:26:45 2023","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":2.8599999999999999,"15min":3.077},"min":{"1min":0.28199999999999997,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":10.695,"5min":14.371,"15min":23.045999999999999},"last":4.2889999999999997},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.1240000000000001,"15min":2.52},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":4.6959999999999997,"5min":11.916,"15min":28.257000000000001},"last":4.3280000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739271,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6196536,"kb_used_data":4284200,"kb_used_omap":580,"kb_used_meta":1911739,"kb_avail":88175304,"statfs":{"total":96636764160,"available":90291511296,"internally_reserved":0,"allocated":4387020800,"data_stored":18164317133,"data_compressed":12381714,"data_compressed_allocated":561618944,"data_compressed_original":13880217600,"omap_allocated":594767,"internal_metadata":1957620913},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,1,0,3,9,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:27:19 2023","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":2.052,"15min":2.1190000000000002},"min":{"1min":0.193,"5min":0.182,"15min":0.182},"max":{"1min":9.5619999999999994,"5min":15.090999999999999,"15min":15.090999999999999},"last":5.6589999999999998},{"interface":"front","average":{"1min":4.0449999999999999,"5min":3.3900000000000001,"15min":3.2490000000000001},"min":{"1min":0.222,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":10.9,"5min":15.265000000000001,"15min":15.265000000000001},"last":2.895}]},{"osd":2,"last update":"Sat Sep 23 08:27:19 2023","interfaces":[{"interface":"back","average":{"1min":3.5649999999999999,"5min":3.5219999999999998,"15min":3.5099999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":12.680999999999999,"5min":32.962000000000003,"15min":32.962000000000003},"last":5.8659999999999997},{"interface":"front","average":{"1min":3.2040000000000002,"5min":3.3919999999999999,"15min":2.8860000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":12.645,"5min":32.088999999999999,"15min":32.088999999999999},"last":5.7199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739271,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6207708,"kb_used_data":4286740,"kb_used_omap":583,"kb_used_meta":1920376,"kb_avail":88164132,"statfs":{"total":96636764160,"available":90280071168,"internally_reserved":0,"allocated":4389621760,"data_stored":18164309323,"data_compressed":12400366,"data_compressed_allocated":561618944,"data_compressed_original":13877964800,"omap_allocated":597990,"internal_metadata":1966465050},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,5,7,1,8,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":1.861,"5min":2.996,"15min":2.5910000000000002},"min":{"1min":0.26400000000000001,"5min":0.187,"15min":0.161},"max":{"1min":5.9199999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":5.0449999999999999},{"interface":"front","average":{"1min":1.7470000000000001,"5min":3.4460000000000002,"15min":3.242},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":5.1050000000000004,"5min":29.521000000000001,"15min":29.521000000000001},"last":5.1920000000000002}]},{"osd":2,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":2.2759999999999998,"5min":3.6779999999999999,"15min":3.6579999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.19},"max":{"1min":12.074999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":3.0449999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.706,"15min":1.6739999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":5.8310000000000004,"5min":11.871,"15min":18.353000000000002},"last":0.21299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4386181120,"data_stored":18163355648,"data_compressed":12378945,"data_compressed_allocated":561582080,"data_compressed_original":13879226368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4388683776,"data_stored":18163351552,"data_compressed":12397374,"data_compressed_allocated":561582080,"data_compressed_original":13876846592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4386390016,"data_stored":18163331072,"data_compressed":11788980,"data_compressed_allocated":561582080,"data_compressed_original":13878996992,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:27:40.147 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:27:40.148 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:28:00.149 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:28:00.749 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:28:00.761 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:28:00.776 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2377,"stamp":"2023-09-23T08:27:59.334914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20335883868,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1188836,"num_read_kb":4415968,"num_write":1114340,"num_write_kb":4399525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78221,"ondisk_log_size":78221,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17847016,"kb_used_data":12888848,"kb_used_omap":1738,"kb_used_meta":4956405,"kb_avail":265268504,"statfs":{"total":289910292480,"available":271634948096,"internally_reserved":0,"allocated":13198180352,"data_stored":54492664216,"data_compressed":36567894,"data_compressed_allocated":1684783104,"data_compressed_original":41605025792,"omap_allocated":1780229,"internal_metadata":5075359227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,11,5,8,17,5,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7311360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2109,"num_read_kb":8412,"num_write":2074,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005503"},"pg_stats":[{"pgid":"2.7","version":"100'5790","reported_seq":9715,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.766780+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:27:53.766780+0000","last_peered":"2023-09-23T08:27:53.766780+0000","last_clean":"2023-09-23T08:27:53.766780+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:27:53.766780+0000","last_undegraded":"2023-09-23T08:27:53.766780+0000","last_fullsized":"2023-09-23T08:27:53.766780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'8426","reported_seq":14516,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.771162+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:27:53.771162+0000","last_peered":"2023-09-23T08:27:53.771162+0000","last_clean":"2023-09-23T08:27:53.771162+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:27:53.771162+0000","last_undegraded":"2023-09-23T08:27:53.771162+0000","last_fullsized":"2023-09-23T08:27:53.771162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8426,"log_dups_size":0,"ondisk_log_size":8426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"100'4105","reported_seq":6736,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.766538+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:27:53.766538+0000","last_peered":"2023-09-23T08:27:53.766538+0000","last_clean":"2023-09-23T08:27:53.766538+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:27:53.766538+0000","last_undegraded":"2023-09-23T08:27:53.766538+0000","last_fullsized":"2023-09-23T08:27:53.766538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'281206","reported_seq":563443,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:55.478448+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:27:55.478448+0000","last_peered":"2023-09-23T08:27:55.478448+0000","last_clean":"2023-09-23T08:27:55.478448+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:27:55.478448+0000","last_undegraded":"2023-09-23T08:27:55.478448+0000","last_fullsized":"2023-09-23T08:27:55.478448+0000","mapping_epoch":18,"log_start":"98'271138","ondisk_log_start":"98'271138","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5198336000,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":283175,"num_read_kb":1110588,"num_write":279824,"num_write_kb":1119296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3156,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.765866+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:27:53.765866+0000","last_peered":"2023-09-23T08:27:53.765866+0000","last_clean":"2023-09-23T08:27:53.765866+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:27:53.765866+0000","last_undegraded":"2023-09-23T08:27:53.765866+0000","last_fullsized":"2023-09-23T08:27:53.765866+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"100'265756","reported_seq":532866,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:27:54.504067+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:27:54.504067+0000","last_peered":"2023-09-23T08:27:54.504067+0000","last_clean":"2023-09-23T08:27:54.504067+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:27:54.504067+0000","last_undegraded":"2023-09-23T08:27:54.504067+0000","last_fullsized":"2023-09-23T08:27:54.504067+0000","mapping_epoch":18,"log_start":"97'255715","ondisk_log_start":"97'255715","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1593,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4926148627,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":268078,"num_read_kb":1051780,"num_write":264455,"num_write_kb":1057814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":273,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:26:55.840802+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:26:55.840802+0000","last_peered":"2023-09-23T08:26:55.840802+0000","last_clean":"2023-09-23T08:26:55.840802+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:26:55.840802+0000","last_undegraded":"2023-09-23T08:26:55.840802+0000","last_fullsized":"2023-09-23T08:26:55.840802+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'6944","reported_seq":10806,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.768684+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:27:53.768684+0000","last_peered":"2023-09-23T08:27:53.768684+0000","last_clean":"2023-09-23T08:27:53.768684+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:27:53.768684+0000","last_undegraded":"2023-09-23T08:27:53.768684+0000","last_fullsized":"2023-09-23T08:27:53.768684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6944,"log_dups_size":0,"ondisk_log_size":6944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"100'267471","reported_seq":536538,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:54.509552+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:27:54.509552+0000","last_peered":"2023-09-23T08:27:54.509552+0000","last_clean":"2023-09-23T08:27:54.509552+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:27:54.509552+0000","last_undegraded":"2023-09-23T08:27:54.509552+0000","last_fullsized":"2023-09-23T08:27:54.509552+0000","mapping_epoch":18,"log_start":"97'257469","ondisk_log_start":"97'257469","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4942086144,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":269959,"num_read_kb":1059108,"num_write":266179,"num_write_kb":1064716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3734,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:48.652264+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:27:48.652264+0000","last_peered":"2023-09-23T08:27:48.652264+0000","last_clean":"2023-09-23T08:27:48.652264+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:27:48.652264+0000","last_undegraded":"2023-09-23T08:27:48.652264+0000","last_fullsized":"2023-09-23T08:27:48.652264+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"100'285541","reported_seq":571711,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:55.467347+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:27:55.467347+0000","last_peered":"2023-09-23T08:27:55.467347+0000","last_clean":"2023-09-23T08:27:55.467347+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:27:55.467347+0000","last_undegraded":"2023-09-23T08:27:55.467347+0000","last_fullsized":"2023-09-23T08:27:55.467347+0000","mapping_epoch":18,"log_start":"98'275522","ondisk_log_start":"98'275522","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5268713472,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":287088,"num_read_kb":1126728,"num_write":284184,"num_write_kb":1136736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"100'4911","reported_seq":7525,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:54.807753+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:27:54.807753+0000","last_peered":"2023-09-23T08:27:54.807753+0000","last_clean":"2023-09-23T08:27:54.807753+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:27:54.807753+0000","last_undegraded":"2023-09-23T08:27:54.807753+0000","last_fullsized":"2023-09-23T08:27:54.807753+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"100'4951","reported_seq":9401,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:53.767008+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:27:53.767008+0000","last_peered":"2023-09-23T08:27:53.767008+0000","last_clean":"2023-09-23T08:27:53.767008+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:27:53.767008+0000","last_undegraded":"2023-09-23T08:27:53.767008+0000","last_fullsized":"2023-09-23T08:27:53.767008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20335284243,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1108300,"num_read_kb":4348204,"num_write":1094642,"num_write_kb":4378562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13195587584,"data_stored":54490185728,"data_compressed":36561767,"data_compressed_allocated":1684709376,"data_compressed_original":41602453504,"omap_allocated":0,"internal_metadata":0},"log_size":40130,"ondisk_log_size":40130,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":37992,"ondisk_log_size":37992,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739278,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6118412,"kb_used_data":4296516,"kb_used_omap":573,"kb_used_meta":1821314,"kb_avail":88253428,"statfs":{"total":96636764160,"available":90371510272,"internally_reserved":0,"allocated":4399632384,"data_stored":18163828864,"data_compressed":11788687,"data_compressed_allocated":561569792,"data_compressed_original":13867581440,"omap_allocated":587472,"internal_metadata":1865025840},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,2,0,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.0489999999999999,"5min":2.6829999999999998,"15min":2.169},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.273,"5min":14.285,"15min":14.285},"last":8.1969999999999992},{"interface":"front","average":{"1min":3.4889999999999999,"5min":2.6040000000000001,"15min":2.25},"min":{"1min":0.29899999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.294,"5min":14.321,"15min":22.266999999999999},"last":5.6520000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.1280000000000001,"5min":2.8010000000000002,"15min":3.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.516,"5min":10.695,"15min":19.433},"last":22.099},{"interface":"front","average":{"1min":3.1549999999999998,"5min":2.1829999999999998,"15min":2.4609999999999999},"min":{"1min":0.32600000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":7.6790000000000003,"5min":11.081,"15min":28.257000000000001},"last":22.135999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739275,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6312720,"kb_used_data":4296328,"kb_used_omap":580,"kb_used_meta":2015803,"kb_avail":88059120,"statfs":{"total":96636764160,"available":90172538880,"internally_reserved":0,"allocated":4399439872,"data_stored":18164444109,"data_compressed":12380110,"data_compressed_allocated":561606656,"data_compressed_original":13868478464,"omap_allocated":594767,"internal_metadata":2064182449},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,8,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:27:19 2023","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":2.052,"15min":2.1190000000000002},"min":{"1min":0.193,"5min":0.182,"15min":0.182},"max":{"1min":9.5619999999999994,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":4.0449999999999999,"5min":3.3900000000000001,"15min":3.2490000000000001},"min":{"1min":0.222,"5min":0.219,"15min":0.16800000000000001},"max":{"1min":10.9,"5min":15.265000000000001,"15min":15.265000000000001},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:27:19 2023","interfaces":[{"interface":"back","average":{"1min":3.5649999999999999,"5min":3.5219999999999998,"15min":3.5099999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":12.680999999999999,"5min":32.962000000000003,"15min":32.962000000000003},"last":1.893},{"interface":"front","average":{"1min":3.2040000000000002,"5min":3.3919999999999999,"15min":2.8860000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":12.645,"5min":32.088999999999999,"15min":32.088999999999999},"last":2.0609999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739274,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5415884,"kb_used_data":4296004,"kb_used_omap":583,"kb_used_meta":1119288,"kb_avail":88955956,"statfs":{"total":96636764160,"available":91090898944,"internally_reserved":0,"allocated":4399108096,"data_stored":18164391243,"data_compressed":12399097,"data_compressed_allocated":561606656,"data_compressed_original":13868965888,"omap_allocated":597990,"internal_metadata":1146150938},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,3,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":1.861,"5min":2.996,"15min":2.5910000000000002},"min":{"1min":0.26400000000000001,"5min":0.187,"15min":0.161},"max":{"1min":5.9199999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.7470000000000001,"5min":3.4460000000000002,"15min":3.242},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":5.1050000000000004,"5min":29.521000000000001,"15min":29.521000000000001},"last":1.0369999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":2.2759999999999998,"5min":3.6779999999999999,"15min":3.6579999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.19},"max":{"1min":12.074999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":7.0880000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.706,"15min":1.6739999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":5.8310000000000004,"5min":11.871,"15min":18.353000000000002},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4398657536,"data_stored":18163376128,"data_compressed":12377052,"data_compressed_allocated":561569792,"data_compressed_original":13867347968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4398116864,"data_stored":18163441664,"data_compressed":12396262,"data_compressed_allocated":561569792,"data_compressed_original":13867917312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4398813184,"data_stored":18163367936,"data_compressed":11788453,"data_compressed_allocated":561569792,"data_compressed_original":13867188224,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:28:00.778 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:28:00.778 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:28:20.779 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:28:21.520 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:28:21.534 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:28:21.548 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2389,"stamp":"2023-09-23T08:28:21.361224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20353635932,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1194435,"num_read_kb":4438292,"num_write":1119454,"num_write_kb":4419981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78413,"ondisk_log_size":78413,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17241728,"kb_used_data":12934560,"kb_used_omap":1738,"kb_used_meta":4305397,"kb_avail":265873792,"statfs":{"total":289910292480,"available":272254763008,"internally_reserved":0,"allocated":13244989440,"data_stored":54493020655,"data_compressed":36562137,"data_compressed_allocated":1684643840,"data_compressed_original":41560338432,"omap_allocated":1780229,"internal_metadata":4408727035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,9,15,19,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10551296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3318,"num_read_kb":13228,"num_write":3058,"num_write_kb":12232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009423"},"pg_stats":[{"pgid":"2.7","version":"101'5814","reported_seq":9756,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.372882+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:28:19.372882+0000","last_peered":"2023-09-23T08:28:19.372882+0000","last_clean":"2023-09-23T08:28:19.372882+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:28:19.372882+0000","last_undegraded":"2023-09-23T08:28:19.372882+0000","last_fullsized":"2023-09-23T08:28:19.372882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'8458","reported_seq":14568,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.372348+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:28:19.372348+0000","last_peered":"2023-09-23T08:28:19.372348+0000","last_clean":"2023-09-23T08:28:19.372348+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:28:19.372348+0000","last_undegraded":"2023-09-23T08:28:19.372348+0000","last_fullsized":"2023-09-23T08:28:19.372348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'4121","reported_seq":6763,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.372557+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:28:19.372557+0000","last_peered":"2023-09-23T08:28:19.372557+0000","last_clean":"2023-09-23T08:28:19.372557+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:28:19.372557+0000","last_undegraded":"2023-09-23T08:28:19.372557+0000","last_fullsized":"2023-09-23T08:28:19.372557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"101'282484","reported_seq":566146,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:21.107232+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:28:21.107232+0000","last_peered":"2023-09-23T08:28:21.107232+0000","last_clean":"2023-09-23T08:28:21.107232+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:28:21.107232+0000","last_undegraded":"2023-09-23T08:28:21.107232+0000","last_fullsized":"2023-09-23T08:28:21.107232+0000","mapping_epoch":18,"log_start":"98'272446","ondisk_log_start":"98'272446","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5202837504,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":284597,"num_read_kb":1116264,"num_write":281102,"num_write_kb":1124408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3167,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.372074+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:28:19.372074+0000","last_peered":"2023-09-23T08:28:19.372074+0000","last_clean":"2023-09-23T08:28:19.372074+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:28:19.372074+0000","last_undegraded":"2023-09-23T08:28:19.372074+0000","last_fullsized":"2023-09-23T08:28:19.372074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'266970","reported_seq":535478,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:28:19.889899+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:28:19.889899+0000","last_peered":"2023-09-23T08:28:19.889899+0000","last_clean":"2023-09-23T08:28:19.889899+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:28:19.889899+0000","last_undegraded":"2023-09-23T08:28:19.889899+0000","last_fullsized":"2023-09-23T08:28:19.889899+0000","mapping_epoch":18,"log_start":"98'256933","ondisk_log_start":"98'256933","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1793,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 239s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4930441235,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":269454,"num_read_kb":1057264,"num_write":265669,"num_write_kb":1062670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:03.281242+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:27:03.281242+0000","last_peered":"2023-09-23T08:27:03.281242+0000","last_clean":"2023-09-23T08:27:03.281242+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:27:03.281242+0000","last_undegraded":"2023-09-23T08:27:03.281242+0000","last_fullsized":"2023-09-23T08:27:03.281242+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'6968","reported_seq":10846,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.372575+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:28:19.372575+0000","last_peered":"2023-09-23T08:28:19.372575+0000","last_clean":"2023-09-23T08:28:19.372575+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:28:19.372575+0000","last_undegraded":"2023-09-23T08:28:19.372575+0000","last_fullsized":"2023-09-23T08:28:19.372575+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"101'268706","reported_seq":539136,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.850980+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:28:19.850980+0000","last_peered":"2023-09-23T08:28:19.850980+0000","last_clean":"2023-09-23T08:28:19.850980+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:28:19.850980+0000","last_undegraded":"2023-09-23T08:28:19.850980+0000","last_fullsized":"2023-09-23T08:28:19.850980+0000","mapping_epoch":18,"log_start":"98'258682","ondisk_log_start":"98'258682","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4946239488,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":271319,"num_read_kb":1064536,"num_write":267414,"num_write_kb":1069656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3740,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:14.249503+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:28:14.249503+0000","last_peered":"2023-09-23T08:28:14.249503+0000","last_clean":"2023-09-23T08:28:14.249503+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:28:14.249503+0000","last_undegraded":"2023-09-23T08:28:14.249503+0000","last_fullsized":"2023-09-23T08:28:14.249503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"101'286928","reported_seq":574537,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:21.142804+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:28:21.142804+0000","last_peered":"2023-09-23T08:28:21.142804+0000","last_clean":"2023-09-23T08:28:21.142804+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:28:21.142804+0000","last_undegraded":"2023-09-23T08:28:21.142804+0000","last_fullsized":"2023-09-23T08:28:21.142804+0000","mapping_epoch":18,"log_start":"98'276841","ondisk_log_start":"98'276841","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5273518080,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":288529,"num_read_kb":1132464,"num_write":285571,"num_write_kb":1142284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"101'4927","reported_seq":7552,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.384570+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:28:19.384570+0000","last_peered":"2023-09-23T08:28:19.384570+0000","last_clean":"2023-09-23T08:28:19.384570+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:28:19.384570+0000","last_undegraded":"2023-09-23T08:28:19.384570+0000","last_fullsized":"2023-09-23T08:28:19.384570+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"101'4975","reported_seq":9441,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:19.373275+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:28:19.373275+0000","last_peered":"2023-09-23T08:28:19.373275+0000","last_clean":"2023-09-23T08:28:19.373275+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:28:19.373275+0000","last_undegraded":"2023-09-23T08:28:19.373275+0000","last_fullsized":"2023-09-23T08:28:19.373275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20353036307,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1113899,"num_read_kb":4370528,"num_write":1099756,"num_write_kb":4399018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13242318848,"data_stored":54490648576,"data_compressed":36556782,"data_compressed_allocated":1684566016,"data_compressed_original":41557925888,"omap_allocated":0,"internal_metadata":0},"log_size":40186,"ondisk_log_size":40186,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38128,"ondisk_log_size":38128,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5467196,"kb_used_data":4311408,"kb_used_omap":573,"kb_used_meta":1155202,"kb_avail":88904644,"statfs":{"total":96636764160,"available":91038355456,"internally_reserved":0,"allocated":4414881792,"data_stored":18163947677,"data_compressed":11787886,"data_compressed_allocated":561524736,"data_compressed_original":13853016064,"omap_allocated":587472,"internal_metadata":1182927152},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,5,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.0489999999999999,"5min":2.6829999999999998,"15min":2.169},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.273,"5min":14.285,"15min":14.285},"last":0.496},{"interface":"front","average":{"1min":3.4889999999999999,"5min":2.6040000000000001,"15min":2.25},"min":{"1min":0.29899999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.294,"5min":14.321,"15min":22.266999999999999},"last":0.251}]},{"osd":1,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.1280000000000001,"5min":2.8010000000000002,"15min":3.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.516,"5min":10.695,"15min":19.433},"last":3.0590000000000002},{"interface":"front","average":{"1min":3.1549999999999998,"5min":2.1829999999999998,"15min":2.4609999999999999},"min":{"1min":0.32600000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":7.6790000000000003,"5min":11.081,"15min":28.257000000000001},"last":3.0950000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739280,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6275532,"kb_used_data":4312068,"kb_used_omap":580,"kb_used_meta":1962875,"kb_avail":88096308,"statfs":{"total":96636764160,"available":90210619392,"internally_reserved":0,"allocated":4415557632,"data_stored":18164534250,"data_compressed":12377541,"data_compressed_allocated":561557504,"data_compressed_original":13853069312,"omap_allocated":594767,"internal_metadata":2009984177},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,9,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":2.0880000000000001,"15min":2.052},"min":{"1min":0.214,"5min":0.185,"15min":0.182},"max":{"1min":6.1500000000000004,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.2610000000000001,"5min":3.3599999999999999,"15min":3.1920000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.215},"max":{"1min":12.536,"5min":15.265000000000001,"15min":15.265000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.5819999999999999,"15min":3.5089999999999999},"min":{"1min":0.27500000000000002,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":15.269,"5min":32.962000000000003,"15min":32.962000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":3.0030000000000001,"5min":3.444,"15min":2.8940000000000001},"min":{"1min":0.28499999999999998,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":13.029999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":0.33000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739279,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5499000,"kb_used_data":4311084,"kb_used_omap":583,"kb_used_meta":1187320,"kb_avail":88872840,"statfs":{"total":96636764160,"available":91005788160,"internally_reserved":0,"allocated":4414550016,"data_stored":18164538728,"data_compressed":12396710,"data_compressed_allocated":561561600,"data_compressed_original":13854253056,"omap_allocated":597990,"internal_metadata":1215815706},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,9,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":1.861,"5min":2.996,"15min":2.5910000000000002},"min":{"1min":0.26400000000000001,"5min":0.187,"15min":0.161},"max":{"1min":5.9199999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.7470000000000001,"5min":3.4460000000000002,"15min":3.242},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":5.1050000000000004,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:27:33 2023","interfaces":[{"interface":"back","average":{"1min":2.2759999999999998,"5min":3.6779999999999999,"15min":3.6579999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.19},"max":{"1min":12.074999999999999,"5min":18.943999999999999,"15min":20.216999999999999},"last":4.8609999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.706,"15min":1.6739999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":5.8310000000000004,"5min":11.871,"15min":18.353000000000002},"last":4.5220000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4414406656,"data_stored":18163576832,"data_compressed":12375023,"data_compressed_allocated":561520640,"data_compressed_original":13852385280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4413698048,"data_stored":18163548160,"data_compressed":12394123,"data_compressed_allocated":561524736,"data_compressed_original":13853028352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4414214144,"data_stored":18163523584,"data_compressed":11787636,"data_compressed_allocated":561520640,"data_compressed_original":13852512256,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:28:21.549 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:28:21.550 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-09-23T08:28:22.195 INFO:teuthology.orchestra.run.smithi140.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-09-23T08:28:22.212 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:28:42.214 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:28:42.917 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:28:42.943 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:28:42.962 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2399,"stamp":"2023-09-23T08:28:41.386050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20366485084,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1198217,"num_read_kb":4453364,"num_write":1123115,"num_write_kb":4434625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78432,"ondisk_log_size":78432,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16393596,"kb_used_data":12969692,"kb_used_omap":1740,"kb_used_meta":3422131,"kb_avail":266721924,"statfs":{"total":289910292480,"available":273123250176,"internally_reserved":0,"allocated":13280964608,"data_stored":54493233647,"data_compressed":36559836,"data_compressed_allocated":1684574208,"data_compressed_original":41526210560,"omap_allocated":1781909,"internal_metadata":3504263019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,3,9,9,17,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9162752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2642,"num_read_kb":10528,"num_write":2603,"num_write_kb":10412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.016863"},"pg_stats":[{"pgid":"2.7","version":"101'5832","reported_seq":9786,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.846931+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:28:39.846931+0000","last_peered":"2023-09-23T08:28:39.846931+0000","last_clean":"2023-09-23T08:28:39.846931+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:28:39.846931+0000","last_undegraded":"2023-09-23T08:28:39.846931+0000","last_fullsized":"2023-09-23T08:28:39.846931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'8482","reported_seq":14609,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.847192+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:28:39.847192+0000","last_peered":"2023-09-23T08:28:39.847192+0000","last_clean":"2023-09-23T08:28:39.847192+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:28:39.847192+0000","last_undegraded":"2023-09-23T08:28:39.847192+0000","last_fullsized":"2023-09-23T08:28:39.847192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'4133","reported_seq":6783,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.847555+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:28:39.847555+0000","last_peered":"2023-09-23T08:28:39.847555+0000","last_clean":"2023-09-23T08:28:39.847555+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:28:39.847555+0000","last_undegraded":"2023-09-23T08:28:39.847555+0000","last_fullsized":"2023-09-23T08:28:39.847555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"101'283288","reported_seq":567780,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:36.617007+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:28:36.617007+0000","last_peered":"2023-09-23T08:28:36.617007+0000","last_clean":"2023-09-23T08:28:36.617007+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:28:36.617007+0000","last_undegraded":"2023-09-23T08:28:36.617007+0000","last_fullsized":"2023-09-23T08:28:36.617007+0000","mapping_epoch":18,"log_start":"98'273253","ondisk_log_start":"98'273253","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5205643264,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":285427,"num_read_kb":1119576,"num_write":281906,"num_write_kb":1127624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3176,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.860082+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:28:39.860082+0000","last_peered":"2023-09-23T08:28:39.860082+0000","last_clean":"2023-09-23T08:28:39.860082+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:28:39.860082+0000","last_undegraded":"2023-09-23T08:28:39.860082+0000","last_fullsized":"2023-09-23T08:28:39.860082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'267979","reported_seq":537519,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:28:40.514750+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:28:40.514750+0000","last_peered":"2023-09-23T08:28:40.514750+0000","last_clean":"2023-09-23T08:28:40.514750+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:28:40.514750+0000","last_undegraded":"2023-09-23T08:28:40.514750+0000","last_fullsized":"2023-09-23T08:28:40.514750+0000","mapping_epoch":18,"log_start":"98'257950","ondisk_log_start":"98'257950","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":1933,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 259s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4934037523,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":270476,"num_read_kb":1061336,"num_write":266678,"num_write_kb":1066706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:03.281242+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:27:03.281242+0000","last_peered":"2023-09-23T08:27:03.281242+0000","last_clean":"2023-09-23T08:27:03.281242+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:27:03.281242+0000","last_undegraded":"2023-09-23T08:27:03.281242+0000","last_fullsized":"2023-09-23T08:27:03.281242+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'6986","reported_seq":10876,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.847411+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:28:39.847411+0000","last_peered":"2023-09-23T08:28:39.847411+0000","last_clean":"2023-09-23T08:28:39.847411+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:28:39.847411+0000","last_undegraded":"2023-09-23T08:28:39.847411+0000","last_fullsized":"2023-09-23T08:28:39.847411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"101'269721","reported_seq":541226,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:40.478707+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:28:40.478707+0000","last_peered":"2023-09-23T08:28:40.478707+0000","last_clean":"2023-09-23T08:28:40.478707+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:28:40.478707+0000","last_undegraded":"2023-09-23T08:28:40.478707+0000","last_fullsized":"2023-09-23T08:28:40.478707+0000","mapping_epoch":18,"log_start":"98'259691","ondisk_log_start":"98'259691","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4949803008,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":272395,"num_read_kb":1068832,"num_write":268429,"num_write_kb":1073716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3744,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:34.735137+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:28:34.735137+0000","last_peered":"2023-09-23T08:28:34.735137+0000","last_clean":"2023-09-23T08:28:34.735137+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:28:34.735137+0000","last_undegraded":"2023-09-23T08:28:34.735137+0000","last_fullsized":"2023-09-23T08:28:34.735137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"101'287761","reported_seq":576229,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:36.640587+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:28:36.640587+0000","last_peered":"2023-09-23T08:28:36.640587+0000","last_clean":"2023-09-23T08:28:36.640587+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:28:36.640587+0000","last_undegraded":"2023-09-23T08:28:36.640587+0000","last_fullsized":"2023-09-23T08:28:36.640587+0000","mapping_epoch":18,"log_start":"98'277746","ondisk_log_start":"98'277746","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5276401664,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":289383,"num_read_kb":1135856,"num_write":286404,"num_write_kb":1145616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"101'4933","reported_seq":7564,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:34.733664+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:28:34.733664+0000","last_peered":"2023-09-23T08:28:34.733664+0000","last_clean":"2023-09-23T08:28:34.733664+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:28:34.733664+0000","last_undegraded":"2023-09-23T08:28:34.733664+0000","last_fullsized":"2023-09-23T08:28:34.733664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"101'4993","reported_seq":9471,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:28:39.847296+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:28:39.847296+0000","last_peered":"2023-09-23T08:28:39.847296+0000","last_clean":"2023-09-23T08:28:39.847296+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:28:39.847296+0000","last_undegraded":"2023-09-23T08:28:39.847296+0000","last_fullsized":"2023-09-23T08:28:39.847296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":4993,"log_dups_size":0,"ondisk_log_size":4993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20365885459,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1117681,"num_read_kb":4385600,"num_write":1103417,"num_write_kb":4413662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13277593600,"data_stored":54490730496,"data_compressed":36553191,"data_compressed_allocated":1684500480,"data_compressed_original":41524350976,"omap_allocated":0,"internal_metadata":0},"log_size":40109,"ondisk_log_size":40109,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38224,"ondisk_log_size":38224,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739287,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5548420,"kb_used_data":4324212,"kb_used_omap":573,"kb_used_meta":1223618,"kb_avail":88823420,"statfs":{"total":96636764160,"available":90955182080,"internally_reserved":0,"allocated":4427993088,"data_stored":18164000925,"data_compressed":11787152,"data_compressed_allocated":561496064,"data_compressed_original":13840551936,"omap_allocated":587472,"internal_metadata":1252985136},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,0,2,3,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.0489999999999999,"5min":2.6829999999999998,"15min":2.169},"min":{"1min":0.29799999999999999,"5min":0.19,"15min":0.17599999999999999},"max":{"1min":11.273,"5min":14.285,"15min":14.285},"last":10.516},{"interface":"front","average":{"1min":3.4889999999999999,"5min":2.6040000000000001,"15min":2.25},"min":{"1min":0.29899999999999999,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":10.294,"5min":14.321,"15min":22.266999999999999},"last":2.048}]},{"osd":1,"last update":"Sat Sep 23 08:27:49 2023","interfaces":[{"interface":"back","average":{"1min":3.1280000000000001,"5min":2.8010000000000002,"15min":3.0190000000000001},"min":{"1min":0.29199999999999998,"5min":0.19400000000000001,"15min":0.189},"max":{"1min":9.516,"5min":10.695,"15min":19.433},"last":0.55000000000000004},{"interface":"front","average":{"1min":3.1549999999999998,"5min":2.1829999999999998,"15min":2.4609999999999999},"min":{"1min":0.32600000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":7.6790000000000003,"5min":11.081,"15min":28.257000000000001},"last":0.58199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739283,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5247188,"kb_used_data":4321164,"kb_used_omap":582,"kb_used_meta":925433,"kb_avail":89124652,"statfs":{"total":96636764160,"available":91263643648,"internally_reserved":0,"allocated":4424871936,"data_stored":18164616170,"data_compressed":12377029,"data_compressed_allocated":561541120,"data_compressed_original":13844262912,"omap_allocated":596447,"internal_metadata":947643937},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,3,7,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":2.0880000000000001,"15min":2.052},"min":{"1min":0.214,"5min":0.185,"15min":0.182},"max":{"1min":6.1500000000000004,"5min":15.090999999999999,"15min":15.090999999999999},"last":6.1509999999999998},{"interface":"front","average":{"1min":2.2610000000000001,"5min":3.3599999999999999,"15min":3.1920000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.215},"max":{"1min":12.536,"5min":15.265000000000001,"15min":15.265000000000001},"last":12.162000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.5819999999999999,"15min":3.5089999999999999},"min":{"1min":0.27500000000000002,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":15.269,"5min":32.962000000000003,"15min":32.962000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":3.0030000000000001,"5min":3.444,"15min":2.8940000000000001},"min":{"1min":0.28499999999999998,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":13.029999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":6.1950000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739283,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5597988,"kb_used_data":4324316,"kb_used_omap":583,"kb_used_meta":1273080,"kb_avail":88773852,"statfs":{"total":96636764160,"available":90904424448,"internally_reserved":0,"allocated":4428099584,"data_stored":18164616552,"data_compressed":12395655,"data_compressed_allocated":561537024,"data_compressed_original":13841395712,"omap_allocated":597990,"internal_metadata":1303633946},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,5,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":2.875,"15min":2.5800000000000001},"min":{"1min":0.19,"5min":0.187,"15min":0.161},"max":{"1min":5.0449999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":3.262,"5min":3.6139999999999999,"15min":3.2629999999999999},"min":{"1min":0.32400000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":13.253,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":3.4369999999999998,"15min":3.6349999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":9.5640000000000001,"5min":16.527000000000001,"15min":20.216999999999999},"last":2.8410000000000002},{"interface":"front","average":{"1min":1.169,"5min":1.569,"15min":1.639},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":8.1129999999999995,"5min":10.224,"15min":18.353000000000002},"last":0.20100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4424073216,"data_stored":18163560448,"data_compressed":12373572,"data_compressed_allocated":561504256,"data_compressed_original":13843156992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4426612736,"data_stored":18163585024,"data_compressed":12392556,"data_compressed_allocated":561500160,"data_compressed_original":13840740352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4426907648,"data_stored":18163585024,"data_compressed":11787063,"data_compressed_allocated":561496064,"data_compressed_original":13840453632,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:28:42.964 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:28:42.964 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:29:02.965 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:29:03.674 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:29:03.699 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:29:03.714 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2410,"stamp":"2023-09-23T08:29:03.405536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20382922332,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1202980,"num_read_kb":4472384,"num_write":1127824,"num_write_kb":4453461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78605,"ondisk_log_size":78605,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16622568,"kb_used_data":13009864,"kb_used_omap":1740,"kb_used_meta":3610931,"kb_avail":266492952,"statfs":{"total":289910292480,"available":272888782848,"internally_reserved":0,"allocated":13322100736,"data_stored":54493344239,"data_compressed":36554333,"data_compressed_allocated":1684426752,"data_compressed_original":41486757888,"omap_allocated":1781909,"internal_metadata":3697594219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,6,19,8,4,14,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8994816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2604,"num_read_kb":10404,"num_write":2600,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004120"},"pg_stats":[{"pgid":"2.7","version":"101'5853","reported_seq":9819,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.331719+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:29:00.331719+0000","last_peered":"2023-09-23T08:29:00.331719+0000","last_clean":"2023-09-23T08:29:00.331719+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:29:00.331719+0000","last_undegraded":"2023-09-23T08:29:00.331719+0000","last_fullsized":"2023-09-23T08:29:00.331719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'8510","reported_seq":14653,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.332265+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:29:00.332265+0000","last_peered":"2023-09-23T08:29:00.332265+0000","last_clean":"2023-09-23T08:29:00.332265+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:29:00.332265+0000","last_undegraded":"2023-09-23T08:29:00.332265+0000","last_fullsized":"2023-09-23T08:29:00.332265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8510,"log_dups_size":0,"ondisk_log_size":8510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'4147","reported_seq":6805,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.331949+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:29:00.331949+0000","last_peered":"2023-09-23T08:29:00.331949+0000","last_clean":"2023-09-23T08:29:00.331949+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:29:00.331949+0000","last_undegraded":"2023-09-23T08:29:00.331949+0000","last_fullsized":"2023-09-23T08:29:00.331949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"101'284577","reported_seq":570406,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:01.860858+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:29:01.860858+0000","last_peered":"2023-09-23T08:29:01.860858+0000","last_clean":"2023-09-23T08:29:01.860858+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:29:01.860858+0000","last_undegraded":"2023-09-23T08:29:01.860858+0000","last_fullsized":"2023-09-23T08:29:01.860858+0000","mapping_epoch":18,"log_start":"99'274576","ondisk_log_start":"99'274576","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5210144768,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":286764,"num_read_kb":1124920,"num_write":283195,"num_write_kb":1132780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3184,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.341654+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:29:00.341654+0000","last_peered":"2023-09-23T08:29:00.341654+0000","last_clean":"2023-09-23T08:29:00.341654+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:29:00.341654+0000","last_undegraded":"2023-09-23T08:29:00.341654+0000","last_fullsized":"2023-09-23T08:29:00.341654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'269004","reported_seq":539574,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:29:00.734309+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:29:00.734309+0000","last_peered":"2023-09-23T08:29:00.734309+0000","last_clean":"2023-09-23T08:29:00.734309+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:29:00.734309+0000","last_undegraded":"2023-09-23T08:29:00.734309+0000","last_fullsized":"2023-09-23T08:29:00.734309+0000","mapping_epoch":18,"log_start":"98'258958","ondisk_log_start":"98'258958","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2034,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 279s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4937682963,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":271496,"num_read_kb":1065408,"num_write":267703,"num_write_kb":1070806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":274,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-09-23T08:27:03.281242+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:27:03.281242+0000","last_peered":"2023-09-23T08:27:03.281242+0000","last_clean":"2023-09-23T08:27:03.281242+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:27:03.281242+0000","last_undegraded":"2023-09-23T08:27:03.281242+0000","last_fullsized":"2023-09-23T08:27:03.281242+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'7007","reported_seq":10909,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.353333+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:29:00.353333+0000","last_peered":"2023-09-23T08:29:00.353333+0000","last_clean":"2023-09-23T08:29:00.353333+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:29:00.353333+0000","last_undegraded":"2023-09-23T08:29:00.353333+0000","last_fullsized":"2023-09-23T08:29:00.353333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":7007,"log_dups_size":0,"ondisk_log_size":7007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"101'270722","reported_seq":543203,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.729118+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:29:00.729118+0000","last_peered":"2023-09-23T08:29:00.729118+0000","last_clean":"2023-09-23T08:29:00.729118+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:29:00.729118+0000","last_undegraded":"2023-09-23T08:29:00.729118+0000","last_fullsized":"2023-09-23T08:29:00.729118+0000","mapping_epoch":18,"log_start":"98'260703","ondisk_log_start":"98'260703","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4953247744,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":273370,"num_read_kb":1072728,"num_write":269430,"num_write_kb":1077720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3749,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.359105+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:29:00.359105+0000","last_peered":"2023-09-23T08:29:00.359105+0000","last_clean":"2023-09-23T08:29:00.359105+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:29:00.359105+0000","last_undegraded":"2023-09-23T08:29:00.359105+0000","last_fullsized":"2023-09-23T08:29:00.359105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"101'289155","reported_seq":579055,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:01.881851+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:29:01.881851+0000","last_peered":"2023-09-23T08:29:01.881851+0000","last_clean":"2023-09-23T08:29:01.881851+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:29:01.881851+0000","last_undegraded":"2023-09-23T08:29:01.881851+0000","last_fullsized":"2023-09-23T08:29:01.881851+0000","mapping_epoch":18,"log_start":"99'279068","ondisk_log_start":"99'279068","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5281247232,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":290814,"num_read_kb":1141564,"num_write":287798,"num_write_kb":1151192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"101'4957","reported_seq":7598,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.996563+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:29:00.996563+0000","last_peered":"2023-09-23T08:29:00.996563+0000","last_clean":"2023-09-23T08:29:00.996563+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:29:00.996563+0000","last_undegraded":"2023-09-23T08:29:00.996563+0000","last_fullsized":"2023-09-23T08:29:00.996563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"101'5014","reported_seq":9504,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:00.331880+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:29:00.331880+0000","last_peered":"2023-09-23T08:29:00.331880+0000","last_clean":"2023-09-23T08:29:00.331880+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:29:00.331880+0000","last_undegraded":"2023-09-23T08:29:00.331880+0000","last_fullsized":"2023-09-23T08:29:00.331880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20382322707,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1122444,"num_read_kb":4404620,"num_write":1108126,"num_write_kb":4432498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13318893568,"data_stored":54490857472,"data_compressed":36547726,"data_compressed_allocated":1684344832,"data_compressed_original":41484738560,"omap_allocated":0,"internal_metadata":0},"log_size":40153,"ondisk_log_size":40153,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38353,"ondisk_log_size":38353,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739291,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5645460,"kb_used_data":4336584,"kb_used_omap":573,"kb_used_meta":1308290,"kb_avail":88726380,"statfs":{"total":96636764160,"available":90855813120,"internally_reserved":0,"allocated":4440662016,"data_stored":18164045981,"data_compressed":11786321,"data_compressed_allocated":561451008,"data_compressed_original":13828407296,"omap_allocated":587472,"internal_metadata":1339689264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":3.7949999999999999,"5min":3.202,"15min":2.3050000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":10.516,"5min":14.285,"15min":14.285},"last":0.249},{"interface":"front","average":{"1min":3.633,"5min":3.0819999999999999,"15min":2.3700000000000001},"min":{"1min":0.224,"5min":0.215,"15min":0.215},"max":{"1min":8.0340000000000007,"5min":14.321,"15min":22.266999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":4.7809999999999997,"5min":3.2879999999999998,"15min":3.214},"min":{"1min":0.313,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":22.099,"5min":22.099,"15min":22.099},"last":1.8240000000000001},{"interface":"front","average":{"1min":4.8179999999999996,"5min":2.7200000000000002,"15min":2.6560000000000001},"min":{"1min":0.34899999999999998,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":22.135999999999999,"5min":22.135999999999999,"15min":28.257000000000001},"last":1.8740000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739288,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5316832,"kb_used_data":4336660,"kb_used_omap":582,"kb_used_meta":979577,"kb_avail":89055008,"statfs":{"total":96636764160,"available":91192328192,"internally_reserved":0,"allocated":4440739840,"data_stored":18164657130,"data_compressed":12374499,"data_compressed_allocated":561487872,"data_compressed_original":13829050368,"omap_allocated":596447,"internal_metadata":1003087393},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,12,2,1,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":2.0880000000000001,"15min":2.052},"min":{"1min":0.214,"5min":0.185,"15min":0.182},"max":{"1min":6.1500000000000004,"5min":15.090999999999999,"15min":15.090999999999999},"last":4.8559999999999999},{"interface":"front","average":{"1min":2.2610000000000001,"5min":3.3599999999999999,"15min":3.1920000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.215},"max":{"1min":12.536,"5min":15.265000000000001,"15min":15.265000000000001},"last":2.4399999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.5819999999999999,"15min":3.5089999999999999},"min":{"1min":0.27500000000000002,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":15.269,"5min":32.962000000000003,"15min":32.962000000000003},"last":1.2549999999999999},{"interface":"front","average":{"1min":3.0030000000000001,"5min":3.444,"15min":2.8940000000000001},"min":{"1min":0.28499999999999998,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":13.029999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":3.8250000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739287,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5660276,"kb_used_data":4336620,"kb_used_omap":583,"kb_used_meta":1323064,"kb_avail":88711564,"statfs":{"total":96636764160,"available":90840641536,"internally_reserved":0,"allocated":4440698880,"data_stored":18164641128,"data_compressed":12393513,"data_compressed_allocated":561487872,"data_compressed_original":13829300224,"omap_allocated":597990,"internal_metadata":1354817562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,5,6,3,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":2.875,"15min":2.5800000000000001},"min":{"1min":0.19,"5min":0.187,"15min":0.161},"max":{"1min":5.0449999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":3.262,"5min":3.6139999999999999,"15min":3.2629999999999999},"min":{"1min":0.32400000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":13.253,"5min":29.521000000000001,"15min":29.521000000000001},"last":0.97899999999999998}]},{"osd":2,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":3.4369999999999998,"15min":3.6349999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":9.5640000000000001,"5min":16.527000000000001,"15min":20.216999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.169,"5min":1.569,"15min":1.639},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":8.1129999999999995,"5min":10.224,"15min":18.353000000000002},"last":0.73599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4440031232,"data_stored":18163617792,"data_compressed":12371054,"data_compressed_allocated":561442816,"data_compressed_original":13827854336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4439318528,"data_stored":18163638272,"data_compressed":12390656,"data_compressed_allocated":561451008,"data_compressed_original":13828567040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4439543808,"data_stored":18163601408,"data_compressed":11786016,"data_compressed_allocated":561451008,"data_compressed_original":13828317184,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:29:03.716 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:29:03.716 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:29:23.717 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:29:24.308 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:29:24.311 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:29:24.334 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2421,"stamp":"2023-09-23T08:29:23.425361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20397352540,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1207208,"num_read_kb":4489284,"num_write":1132017,"num_write_kb":4470233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78743,"ondisk_log_size":78743,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16323872,"kb_used_data":13047304,"kb_used_omap":1740,"kb_used_meta":3274803,"kb_avail":266791648,"statfs":{"total":289910292480,"available":273194647552,"internally_reserved":0,"allocated":13360439296,"data_stored":54493614533,"data_compressed":36550605,"data_compressed_allocated":1684353024,"data_compressed_original":41450524672,"omap_allocated":1781909,"internal_metadata":3353399147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,8,24,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7204864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2198,"num_read_kb":8784,"num_write":2099,"num_write_kb":8396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003353"},"pg_stats":[{"pgid":"2.7","version":"102'5871","reported_seq":9850,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.813808+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:29:20.813808+0000","last_peered":"2023-09-23T08:29:20.813808+0000","last_clean":"2023-09-23T08:29:20.813808+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:29:20.813808+0000","last_undegraded":"2023-09-23T08:29:20.813808+0000","last_fullsized":"2023-09-23T08:29:20.813808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'8534","reported_seq":14694,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.811412+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:29:20.811412+0000","last_peered":"2023-09-23T08:29:20.811412+0000","last_clean":"2023-09-23T08:29:20.811412+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:29:20.811412+0000","last_undegraded":"2023-09-23T08:29:20.811412+0000","last_fullsized":"2023-09-23T08:29:20.811412+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'4159","reported_seq":6826,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.811064+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:29:20.811064+0000","last_peered":"2023-09-23T08:29:20.811064+0000","last_clean":"2023-09-23T08:29:20.811064+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:29:20.811064+0000","last_undegraded":"2023-09-23T08:29:20.811064+0000","last_fullsized":"2023-09-23T08:29:20.811064+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"102'285624","reported_seq":572553,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:22.141071+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:29:22.141071+0000","last_peered":"2023-09-23T08:29:22.141071+0000","last_clean":"2023-09-23T08:29:22.141071+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:29:22.141071+0000","last_undegraded":"2023-09-23T08:29:22.141071+0000","last_fullsized":"2023-09-23T08:29:22.141071+0000","mapping_epoch":18,"log_start":"99'275589","ondisk_log_start":"99'275589","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5213782016,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":287864,"num_read_kb":1129320,"num_write":284242,"num_write_kb":1136968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3193,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.812557+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:29:20.812557+0000","last_peered":"2023-09-23T08:29:20.812557+0000","last_clean":"2023-09-23T08:29:20.812557+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:29:20.812557+0000","last_undegraded":"2023-09-23T08:29:20.812557+0000","last_fullsized":"2023-09-23T08:29:20.812557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'270045","reported_seq":541665,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:29:21.061682+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:29:21.061682+0000","last_peered":"2023-09-23T08:29:21.061682+0000","last_clean":"2023-09-23T08:29:21.061682+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:29:21.061682+0000","last_undegraded":"2023-09-23T08:29:21.061682+0000","last_fullsized":"2023-09-23T08:29:21.061682+0000","mapping_epoch":18,"log_start":"99'259980","ondisk_log_start":"99'259980","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2124,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 300s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4941320211,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":272536,"num_read_kb":1069560,"num_write":268744,"num_write_kb":1074970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:05.828333+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:29:05.828333+0000","last_peered":"2023-09-23T08:29:05.828333+0000","last_clean":"2023-09-23T08:29:05.828333+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:29:05.828333+0000","last_undegraded":"2023-09-23T08:29:05.828333+0000","last_fullsized":"2023-09-23T08:29:05.828333+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'7025","reported_seq":10940,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.810271+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:29:20.810271+0000","last_peered":"2023-09-23T08:29:20.810271+0000","last_clean":"2023-09-23T08:29:20.810271+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:29:20.810271+0000","last_undegraded":"2023-09-23T08:29:20.810271+0000","last_fullsized":"2023-09-23T08:29:20.810271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'271747","reported_seq":545244,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:21.052947+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:29:21.052947+0000","last_peered":"2023-09-23T08:29:21.052947+0000","last_clean":"2023-09-23T08:29:21.052947+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:29:21.052947+0000","last_undegraded":"2023-09-23T08:29:21.052947+0000","last_fullsized":"2023-09-23T08:29:21.052947+0000","mapping_epoch":18,"log_start":"99'261715","ondisk_log_start":"99'261715","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4956655616,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":274385,"num_read_kb":1076784,"num_write":270455,"num_write_kb":1081820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3754,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:16.005186+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:29:16.005186+0000","last_peered":"2023-09-23T08:29:16.005186+0000","last_clean":"2023-09-23T08:29:16.005186+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:29:16.005186+0000","last_undegraded":"2023-09-23T08:29:16.005186+0000","last_fullsized":"2023-09-23T08:29:16.005186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"102'290235","reported_seq":581210,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:22.114337+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:29:22.114337+0000","last_peered":"2023-09-23T08:29:22.114337+0000","last_clean":"2023-09-23T08:29:22.114337+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:29:22.114337+0000","last_undegraded":"2023-09-23T08:29:22.114337+0000","last_fullsized":"2023-09-23T08:29:22.114337+0000","mapping_epoch":18,"log_start":"99'280179","ondisk_log_start":"99'280179","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5284995072,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":291887,"num_read_kb":1145856,"num_write":288878,"num_write_kb":1155512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"102'4970","reported_seq":7620,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.866850+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:29:20.866850+0000","last_peered":"2023-09-23T08:29:20.866850+0000","last_clean":"2023-09-23T08:29:20.866850+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:29:20.866850+0000","last_undegraded":"2023-09-23T08:29:20.866850+0000","last_fullsized":"2023-09-23T08:29:20.866850+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"102'5032","reported_seq":9535,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:20.810847+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:29:20.810847+0000","last_peered":"2023-09-23T08:29:20.810847+0000","last_clean":"2023-09-23T08:29:20.810847+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:29:20.810847+0000","last_undegraded":"2023-09-23T08:29:20.810847+0000","last_fullsized":"2023-09-23T08:29:20.810847+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20396752915,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1126672,"num_read_kb":4421520,"num_write":1112319,"num_write_kb":4449270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13356908544,"data_stored":54491189248,"data_compressed":36544980,"data_compressed_allocated":1684279296,"data_compressed_original":41448890368,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38456,"ondisk_log_size":38456,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739295,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5761756,"kb_used_data":4348820,"kb_used_omap":573,"kb_used_meta":1412354,"kb_avail":88610084,"statfs":{"total":96636764160,"available":90736726016,"internally_reserved":0,"allocated":4453191680,"data_stored":18164136079,"data_compressed":11784969,"data_compressed_allocated":561426432,"data_compressed_original":13816553472,"omap_allocated":587472,"internal_metadata":1446250800},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":3.7949999999999999,"5min":3.202,"15min":2.3050000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":10.516,"5min":14.285,"15min":14.285},"last":0.309},{"interface":"front","average":{"1min":3.633,"5min":3.0819999999999999,"15min":2.3700000000000001},"min":{"1min":0.224,"5min":0.215,"15min":0.215},"max":{"1min":8.0340000000000007,"5min":14.321,"15min":22.266999999999999},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":4.7809999999999997,"5min":3.2879999999999998,"15min":3.214},"min":{"1min":0.313,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":22.099,"5min":22.099,"15min":22.099},"last":0.63900000000000001},{"interface":"front","average":{"1min":4.8179999999999996,"5min":2.7200000000000002,"15min":2.6560000000000001},"min":{"1min":0.34899999999999998,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":22.135999999999999,"5min":22.135999999999999,"15min":28.257000000000001},"last":0.67700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739292,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4803480,"kb_used_data":4349584,"kb_used_omap":582,"kb_used_meta":453305,"kb_avail":89568360,"statfs":{"total":96636764160,"available":91718000640,"internally_reserved":0,"allocated":4453974016,"data_stored":18164739036,"data_compressed":12373274,"data_compressed_allocated":561463296,"data_compressed_original":13816561664,"omap_allocated":596447,"internal_metadata":464184865},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":2.0880000000000001,"15min":2.052},"min":{"1min":0.214,"5min":0.185,"15min":0.182},"max":{"1min":6.1500000000000004,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.307},{"interface":"front","average":{"1min":2.2610000000000001,"5min":3.3599999999999999,"15min":3.1920000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.215},"max":{"1min":12.536,"5min":15.265000000000001,"15min":15.265000000000001},"last":4.0229999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:28:20 2023","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.5819999999999999,"15min":3.5089999999999999},"min":{"1min":0.27500000000000002,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":15.269,"5min":32.962000000000003,"15min":32.962000000000003},"last":3.9260000000000002},{"interface":"front","average":{"1min":3.0030000000000001,"5min":3.444,"15min":2.8940000000000001},"min":{"1min":0.28499999999999998,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":13.029999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":3.4460000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739291,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5758636,"kb_used_data":4348900,"kb_used_omap":583,"kb_used_meta":1409144,"kb_avail":88613204,"statfs":{"total":96636764160,"available":90739920896,"internally_reserved":0,"allocated":4453273600,"data_stored":18164739418,"data_compressed":12392362,"data_compressed_allocated":561463296,"data_compressed_original":13817409536,"omap_allocated":597990,"internal_metadata":1442963482},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,5,22,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":2.875,"15min":2.5800000000000001},"min":{"1min":0.19,"5min":0.187,"15min":0.161},"max":{"1min":5.0449999999999999,"5min":29.236999999999998,"15min":29.236999999999998},"last":3.1280000000000001},{"interface":"front","average":{"1min":3.262,"5min":3.6139999999999999,"15min":3.2629999999999999},"min":{"1min":0.32400000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":13.253,"5min":29.521000000000001,"15min":29.521000000000001},"last":3.7109999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":3.4369999999999998,"15min":3.6349999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19},"max":{"1min":9.5640000000000001,"5min":16.527000000000001,"15min":20.216999999999999},"last":3.6749999999999998},{"interface":"front","average":{"1min":1.169,"5min":1.569,"15min":1.639},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":8.1129999999999995,"5min":10.224,"15min":18.353000000000002},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4452687872,"data_stored":18163716096,"data_compressed":12370273,"data_compressed_allocated":561426432,"data_compressed_original":13815934976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4452040704,"data_stored":18163732480,"data_compressed":12389486,"data_compressed_allocated":561426432,"data_compressed_original":13816537088,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4452179968,"data_stored":18163740672,"data_compressed":11785221,"data_compressed_allocated":561426432,"data_compressed_original":13816418304,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:29:24.335 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:29:24.336 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:29:44.337 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:29:45.004 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:29:45.018 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:29:45.033 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2431,"stamp":"2023-09-23T08:29:43.449690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20412061276,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1211272,"num_read_kb":4505528,"num_write":1136210,"num_write_kb":4487005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78770,"ondisk_log_size":78770,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16545708,"kb_used_data":13083156,"kb_used_omap":1740,"kb_used_meta":3460787,"kb_avail":266569812,"statfs":{"total":289910292480,"available":272967487488,"internally_reserved":0,"allocated":13397151744,"data_stored":54493807045,"data_compressed":36548222,"data_compressed_allocated":1684303872,"data_compressed_original":41415778304,"omap_allocated":1781909,"internal_metadata":3543846763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,6,61,29,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9293824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2583,"num_read_kb":10324,"num_write":2646,"num_write_kb":10584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.020942"},"pg_stats":[{"pgid":"2.7","version":"102'5889","reported_seq":9880,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.287201+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:29:41.287201+0000","last_peered":"2023-09-23T08:29:41.287201+0000","last_clean":"2023-09-23T08:29:41.287201+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:29:41.287201+0000","last_undegraded":"2023-09-23T08:29:41.287201+0000","last_fullsized":"2023-09-23T08:29:41.287201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'8558","reported_seq":14734,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.287107+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:29:41.287107+0000","last_peered":"2023-09-23T08:29:41.287107+0000","last_clean":"2023-09-23T08:29:41.287107+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:29:41.287107+0000","last_undegraded":"2023-09-23T08:29:41.287107+0000","last_fullsized":"2023-09-23T08:29:41.287107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8558,"log_dups_size":0,"ondisk_log_size":8558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'4171","reported_seq":6846,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.286845+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:29:41.286845+0000","last_peered":"2023-09-23T08:29:41.286845+0000","last_clean":"2023-09-23T08:29:41.286845+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:29:41.286845+0000","last_undegraded":"2023-09-23T08:29:41.286845+0000","last_fullsized":"2023-09-23T08:29:41.286845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"102'286712","reported_seq":574730,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:42.637394+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:29:42.637394+0000","last_peered":"2023-09-23T08:29:42.637394+0000","last_clean":"2023-09-23T08:29:42.637394+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:29:42.637394+0000","last_undegraded":"2023-09-23T08:29:42.637394+0000","last_fullsized":"2023-09-23T08:29:42.637394+0000","mapping_epoch":18,"log_start":"99'276710","ondisk_log_start":"99'276710","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5217681408,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":288956,"num_read_kb":1133680,"num_write":285330,"num_write_kb":1141320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3201,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.339724+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:29:41.339724+0000","last_peered":"2023-09-23T08:29:41.339724+0000","last_clean":"2023-09-23T08:29:41.339724+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:29:41.339724+0000","last_undegraded":"2023-09-23T08:29:41.339724+0000","last_fullsized":"2023-09-23T08:29:41.339724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'271069","reported_seq":543677,"reported_epoch":102,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T08:29:41.510859+0000","last_change":"2023-09-23T08:24:20.850612+0000","last_active":"2023-09-23T08:29:41.510859+0000","last_peered":"2023-09-23T08:29:41.510859+0000","last_clean":"2023-09-23T08:29:41.510859+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:29:41.510859+0000","last_undegraded":"2023-09-23T08:29:41.510859+0000","last_fullsized":"2023-09-23T08:29:41.510859+0000","mapping_epoch":18,"log_start":"99'260993","ondisk_log_start":"99'260993","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T07:12:13.725494+0000","last_clean_scrub_stamp":"2023-09-23T07:12:13.725494+0000","objects_scrubbed":2242,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 320s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4944973843,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":273509,"num_read_kb":1073452,"num_write":269768,"num_write_kb":1079066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:05.828333+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:29:05.828333+0000","last_peered":"2023-09-23T08:29:05.828333+0000","last_clean":"2023-09-23T08:29:05.828333+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:29:05.828333+0000","last_undegraded":"2023-09-23T08:29:05.828333+0000","last_fullsized":"2023-09-23T08:29:05.828333+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'7043","reported_seq":10967,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.287102+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:29:41.287102+0000","last_peered":"2023-09-23T08:29:41.287102+0000","last_clean":"2023-09-23T08:29:41.287102+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:29:41.287102+0000","last_undegraded":"2023-09-23T08:29:41.287102+0000","last_fullsized":"2023-09-23T08:29:41.287102+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'272738","reported_seq":547230,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.468172+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:29:41.468172+0000","last_peered":"2023-09-23T08:29:41.468172+0000","last_clean":"2023-09-23T08:29:41.468172+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:29:41.468172+0000","last_undegraded":"2023-09-23T08:29:41.468172+0000","last_fullsized":"2023-09-23T08:29:41.468172+0000","mapping_epoch":18,"log_start":"99'262731","ondisk_log_start":"99'262731","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4960010240,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":275380,"num_read_kb":1080764,"num_write":271446,"num_write_kb":1085784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3758,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:36.170310+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:29:36.170310+0000","last_peered":"2023-09-23T08:29:36.170310+0000","last_clean":"2023-09-23T08:29:36.170310+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:29:36.170310+0000","last_undegraded":"2023-09-23T08:29:36.170310+0000","last_fullsized":"2023-09-23T08:29:36.170310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"102'291325","reported_seq":583302,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:42.668875+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:29:42.668875+0000","last_peered":"2023-09-23T08:29:42.668875+0000","last_clean":"2023-09-23T08:29:42.668875+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:29:42.668875+0000","last_undegraded":"2023-09-23T08:29:42.668875+0000","last_fullsized":"2023-09-23T08:29:42.668875+0000","mapping_epoch":18,"log_start":"99'281301","ondisk_log_start":"99'281301","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5288796160,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":292891,"num_read_kb":1149868,"num_write":289968,"num_write_kb":1159872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"102'4986","reported_seq":7645,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.308230+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:29:41.308230+0000","last_peered":"2023-09-23T08:29:41.308230+0000","last_clean":"2023-09-23T08:29:41.308230+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:29:41.308230+0000","last_undegraded":"2023-09-23T08:29:41.308230+0000","last_fullsized":"2023-09-23T08:29:41.308230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"102'5050","reported_seq":9565,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:41.286987+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:29:41.286987+0000","last_peered":"2023-09-23T08:29:41.286987+0000","last_clean":"2023-09-23T08:29:41.286987+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:29:41.286987+0000","last_undegraded":"2023-09-23T08:29:41.286987+0000","last_fullsized":"2023-09-23T08:29:41.286987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20411461651,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1130736,"num_read_kb":4437764,"num_write":1116512,"num_write_kb":4466042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13393866752,"data_stored":54491369472,"data_compressed":36541940,"data_compressed_allocated":1684230144,"data_compressed_original":41413869568,"omap_allocated":0,"internal_metadata":0},"log_size":40109,"ondisk_log_size":40109,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38562,"ondisk_log_size":38562,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739299,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5805564,"kb_used_data":4360692,"kb_used_omap":573,"kb_used_meta":1444290,"kb_avail":88566276,"statfs":{"total":96636764160,"available":90691866624,"internally_reserved":0,"allocated":4465348608,"data_stored":18164197519,"data_compressed":11784011,"data_compressed_allocated":561410048,"data_compressed_original":13805051904,"omap_allocated":587472,"internal_metadata":1478953264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":3.7949999999999999,"5min":3.202,"15min":2.3050000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":10.516,"5min":14.285,"15min":14.285},"last":0.38100000000000001},{"interface":"front","average":{"1min":3.633,"5min":3.0819999999999999,"15min":2.3700000000000001},"min":{"1min":0.224,"5min":0.215,"15min":0.215},"max":{"1min":8.0340000000000007,"5min":14.321,"15min":22.266999999999999},"last":4.4050000000000002}]},{"osd":1,"last update":"Sat Sep 23 08:28:50 2023","interfaces":[{"interface":"back","average":{"1min":4.7809999999999997,"5min":3.2879999999999998,"15min":3.214},"min":{"1min":0.313,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":22.099,"5min":22.099,"15min":22.099},"last":0.27000000000000002},{"interface":"front","average":{"1min":4.8179999999999996,"5min":2.7200000000000002,"15min":2.6560000000000001},"min":{"1min":0.34899999999999998,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":22.135999999999999,"5min":22.135999999999999,"15min":28.257000000000001},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739296,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4919508,"kb_used_data":4361548,"kb_used_omap":582,"kb_used_meta":557369,"kb_avail":89452332,"statfs":{"total":96636764160,"available":91599187968,"internally_reserved":0,"allocated":4466225152,"data_stored":18164816860,"data_compressed":12372401,"data_compressed_allocated":561446912,"data_compressed_original":13804961792,"omap_allocated":596447,"internal_metadata":570746401},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,9,21,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:29:24 2023","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.1019999999999999,"15min":2.0990000000000002},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.182},"max":{"1min":7.492,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.251},{"interface":"front","average":{"1min":3.7869999999999999,"5min":3.4830000000000001,"15min":3.2469999999999999},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.219},"max":{"1min":13.824,"5min":15.265000000000001,"15min":15.265000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:29:24 2023","interfaces":[{"interface":"back","average":{"1min":3.4820000000000002,"5min":3.9609999999999999,"15min":3.4500000000000002},"min":{"1min":0.34200000000000003,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":16.864999999999998,"5min":32.962000000000003,"15min":32.962000000000003},"last":0.37},{"interface":"front","average":{"1min":2.2130000000000001,"5min":3.4289999999999998,"15min":2.9350000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":7.8959999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739295,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5820636,"kb_used_data":4360916,"kb_used_omap":583,"kb_used_meta":1459128,"kb_avail":88551204,"statfs":{"total":96636764160,"available":90676432896,"internally_reserved":0,"allocated":4465577984,"data_stored":18164792666,"data_compressed":12391810,"data_compressed_allocated":561446912,"data_compressed_original":13805764608,"omap_allocated":597990,"internal_metadata":1494147098},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,47,8,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":2.5800000000000001,"15min":2.5920000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":15.004,"5min":15.004,"15min":29.236999999999998},"last":3.2349999999999999},{"interface":"front","average":{"1min":3.3959999999999999,"5min":3.2629999999999999,"15min":3.2149999999999999},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":9.5060000000000002,"5min":14.664999999999999,"15min":29.521000000000001},"last":3.3460000000000001}]},{"osd":2,"last update":"Sat Sep 23 08:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":3.2730000000000001,"15min":3.5470000000000002},"min":{"1min":0.27600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":12.513999999999999,"5min":14.301,"15min":20.216999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.3,"5min":1.4870000000000001,"15min":1.6679999999999999},"min":{"1min":0.23999999999999999,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":7.1079999999999997,"5min":8.4329999999999998,"15min":18.353000000000002},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4465139712,"data_stored":18163798016,"data_compressed":12369305,"data_compressed_allocated":561410048,"data_compressed_original":13804154880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4464402432,"data_stored":18163781632,"data_compressed":12388698,"data_compressed_allocated":561410048,"data_compressed_original":13804818432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4464324608,"data_stored":18163789824,"data_compressed":11783937,"data_compressed_allocated":561410048,"data_compressed_original":13804896256,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:29:45.034 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T07:12:13.725494+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=7, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=8, tm_min=18, tm_sec=44, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T08:29:45.035 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T08:30:05.035 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T08:30:05.696 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-09-23T08:30:05.729 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-09-23T08:30:05.761 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":2442,"stamp":"2023-09-23T08:30:05.465775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20427097692,"num_objects":9492,"num_object_clones":5332,"num_object_copies":28317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":1215802,"num_read_kb":4523572,"num_write":1140461,"num_write_kb":4504009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79034,"ondisk_log_size":79034,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16784408,"kb_used_data":13119676,"kb_used_omap":1740,"kb_used_meta":3662963,"kb_avail":266331112,"statfs":{"total":289910292480,"available":272723058688,"internally_reserved":0,"allocated":13434548224,"data_stored":54493991365,"data_compressed":36545126,"data_compressed_allocated":1684217856,"data_compressed_original":41380192256,"omap_allocated":1781909,"internal_metadata":3750874987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,10,17,9,13,15,7,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7618560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2343,"num_read_kb":9340,"num_write":2163,"num_write_kb":8652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009701"},"pg_stats":[{"pgid":"2.7","version":"102'5925","reported_seq":9928,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.770908+0000","last_change":"2023-09-23T08:18:56.442842+0000","last_active":"2023-09-23T08:30:01.770908+0000","last_peered":"2023-09-23T08:30:01.770908+0000","last_clean":"2023-09-23T08:30:01.770908+0000","last_became_active":"2023-09-23T07:12:04.895474+0000","last_became_peered":"2023-09-23T07:12:04.895474+0000","last_unstale":"2023-09-23T08:30:01.770908+0000","last_undegraded":"2023-09-23T08:30:01.770908+0000","last_fullsized":"2023-09-23T08:30:01.770908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'5311","last_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_deep_scrub":"93'5311","last_deep_scrub_stamp":"2023-09-23T08:18:56.442637+0000","last_clean_scrub_stamp":"2023-09-23T08:18:56.442637+0000","objects_scrubbed":19,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:47:45.275370+0000","scrub_duration":0.131714202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1072,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8184,"num_read_kb":6875,"num_write":1709,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"102'8606","reported_seq":14798,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.772336+0000","last_change":"2023-09-23T08:19:01.472159+0000","last_active":"2023-09-23T08:30:01.772336+0000","last_peered":"2023-09-23T08:30:01.772336+0000","last_clean":"2023-09-23T08:30:01.772336+0000","last_became_active":"2023-09-23T07:12:04.899230+0000","last_became_peered":"2023-09-23T07:12:04.899230+0000","last_unstale":"2023-09-23T08:30:01.772336+0000","last_undegraded":"2023-09-23T08:30:01.772336+0000","last_fullsized":"2023-09-23T08:30:01.772336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'7788","last_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_deep_scrub":"93'7788","last_deep_scrub_stamp":"2023-09-23T08:19:01.471833+0000","last_clean_scrub_stamp":"2023-09-23T08:19:01.471833+0000","objects_scrubbed":24,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:52:12.255955+0000","scrub_duration":0.197065984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1256,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16682,"num_read_kb":14264,"num_write":3765,"num_write_kb":3678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3483,"num_omap_keys":87,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'4195","reported_seq":6878,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.770948+0000","last_change":"2023-09-23T08:18:53.402344+0000","last_active":"2023-09-23T08:30:01.770948+0000","last_peered":"2023-09-23T08:30:01.770948+0000","last_clean":"2023-09-23T08:30:01.770948+0000","last_became_active":"2023-09-23T07:12:04.895439+0000","last_became_peered":"2023-09-23T07:12:04.895439+0000","last_unstale":"2023-09-23T08:30:01.770948+0000","last_undegraded":"2023-09-23T08:30:01.770948+0000","last_fullsized":"2023-09-23T08:30:01.770948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3784","last_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_deep_scrub":"93'3784","last_deep_scrub_stamp":"2023-09-23T08:18:53.402122+0000","last_clean_scrub_stamp":"2023-09-23T08:18:53.402122+0000","objects_scrubbed":15,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:51:17.225887+0000","scrub_duration":0.11642931199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":837,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6991,"num_read_kb":5846,"num_write":1662,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1252,"num_omap_keys":23,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"102'287794","reported_seq":576977,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:02.984364+0000","last_change":"2023-09-23T08:25:59.595075+0000","last_active":"2023-09-23T08:30:02.984364+0000","last_peered":"2023-09-23T08:30:02.984364+0000","last_clean":"2023-09-23T08:30:02.984364+0000","last_became_active":"2023-09-23T07:12:14.741504+0000","last_became_peered":"2023-09-23T07:12:14.741504+0000","last_unstale":"2023-09-23T08:30:02.984364+0000","last_undegraded":"2023-09-23T08:30:02.984364+0000","last_fullsized":"2023-09-23T08:30:02.984364+0000","mapping_epoch":18,"log_start":"99'277733","ondisk_log_start":"99'277733","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'275063","last_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_deep_scrub":"99'275063","last_deep_scrub_stamp":"2023-09-23T08:25:59.594379+0000","last_clean_scrub_stamp":"2023-09-23T08:25:59.594379+0000","objects_scrubbed":2405,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:13:45.118025+0000","scrub_duration":423.13601427100002,"objects_trimmed":0,"snaptrim_duration":0.022685853999999998,"stat_sum":{"num_bytes":5221466112,"num_objects":2405,"num_object_clones":1382,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":290115,"num_read_kb":1138284,"num_write":286412,"num_write_kb":1145648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'530","reported_seq":3207,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:56.654020+0000","last_change":"2023-09-23T08:18:49.449440+0000","last_active":"2023-09-23T08:29:56.654020+0000","last_peered":"2023-09-23T08:29:56.654020+0000","last_clean":"2023-09-23T08:29:56.654020+0000","last_became_active":"2023-09-23T07:12:04.898237+0000","last_became_peered":"2023-09-23T07:12:04.898237+0000","last_unstale":"2023-09-23T08:29:56.654020+0000","last_undegraded":"2023-09-23T08:29:56.654020+0000","last_fullsized":"2023-09-23T08:29:56.654020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'530","last_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_deep_scrub":"93'530","last_deep_scrub_stamp":"2023-09-23T08:18:49.449196+0000","last_clean_scrub_stamp":"2023-09-23T08:18:49.449196+0000","objects_scrubbed":21,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:00:37.325819+0000","scrub_duration":0.247317649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4580,"num_read_kb":3841,"num_write":862,"num_write_kb":829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1253,"num_omap_keys":23,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"102'272089","reported_seq":545749,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.805206+0000","last_change":"2023-09-23T08:29:44.019214+0000","last_active":"2023-09-23T08:30:01.805206+0000","last_peered":"2023-09-23T08:30:01.805206+0000","last_clean":"2023-09-23T08:30:01.805206+0000","last_became_active":"2023-09-23T07:12:14.742566+0000","last_became_peered":"2023-09-23T07:12:14.742566+0000","last_unstale":"2023-09-23T08:30:01.805206+0000","last_undegraded":"2023-09-23T08:30:01.805206+0000","last_fullsized":"2023-09-23T08:30:01.805206+0000","mapping_epoch":18,"log_start":"99'262014","ondisk_log_start":"99'262014","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'271187","last_scrub_stamp":"2023-09-23T08:29:44.019005+0000","last_deep_scrub":"102'271187","last_deep_scrub_stamp":"2023-09-23T08:29:44.019005+0000","last_clean_scrub_stamp":"2023-09-23T08:29:44.019005+0000","objects_scrubbed":2270,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":324,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T17:52:39.342279+0000","scrub_duration":323.153709354,"objects_trimmed":0,"snaptrim_duration":0.033456026,"stat_sum":{"num_bytes":4948611091,"num_objects":2270,"num_object_clones":1301,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":274554,"num_read_kb":1077604,"num_write":270788,"num_write_kb":1083146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"93'99","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-09-23T08:29:05.828333+0000","last_change":"2023-09-23T08:18:51.444335+0000","last_active":"2023-09-23T08:29:05.828333+0000","last_peered":"2023-09-23T08:29:05.828333+0000","last_clean":"2023-09-23T08:29:05.828333+0000","last_became_active":"2023-09-23T07:12:00.033886+0000","last_became_peered":"2023-09-23T07:12:00.033886+0000","last_unstale":"2023-09-23T08:29:05.828333+0000","last_undegraded":"2023-09-23T08:29:05.828333+0000","last_fullsized":"2023-09-23T08:29:05.828333+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'99","last_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_deep_scrub":"93'99","last_deep_scrub_stamp":"2023-09-23T08:18:51.444143+0000","last_clean_scrub_stamp":"2023-09-23T08:18:51.444143+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T20:02:16.223529+0000","scrub_duration":0.108290944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"102'7079","reported_seq":11015,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.770340+0000","last_change":"2023-09-23T08:18:57.829489+0000","last_active":"2023-09-23T08:30:01.770340+0000","last_peered":"2023-09-23T08:30:01.770340+0000","last_clean":"2023-09-23T08:30:01.770340+0000","last_became_active":"2023-09-23T07:12:04.897955+0000","last_became_peered":"2023-09-23T07:12:04.897955+0000","last_unstale":"2023-09-23T08:30:01.770340+0000","last_undegraded":"2023-09-23T08:30:01.770340+0000","last_fullsized":"2023-09-23T08:30:01.770340+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'6470","last_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_deep_scrub":"93'6470","last_deep_scrub_stamp":"2023-09-23T08:18:57.829282+0000","last_clean_scrub_stamp":"2023-09-23T08:18:57.829282+0000","objects_scrubbed":18,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:40:37.410615+0000","scrub_duration":0.48750004699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1163,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":15280,"num_read_kb":12772,"num_write":4106,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1771,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'273790","reported_seq":549438,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.774981+0000","last_change":"2023-09-23T08:24:18.916012+0000","last_active":"2023-09-23T08:30:01.774981+0000","last_peered":"2023-09-23T08:30:01.774981+0000","last_clean":"2023-09-23T08:30:01.774981+0000","last_became_active":"2023-09-23T07:12:14.742442+0000","last_became_peered":"2023-09-23T07:12:14.742442+0000","last_unstale":"2023-09-23T08:30:01.774981+0000","last_undegraded":"2023-09-23T08:30:01.774981+0000","last_fullsized":"2023-09-23T08:30:01.774981+0000","mapping_epoch":18,"log_start":"99'263748","ondisk_log_start":"99'263748","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'256925","last_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_deep_scrub":"97'256925","last_deep_scrub_stamp":"2023-09-23T08:24:18.915838+0000","last_clean_scrub_stamp":"2023-09-23T08:24:18.915838+0000","objects_scrubbed":2266,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":313,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:17:04.267035+0000","scrub_duration":312.42515482900001,"objects_trimmed":0,"snaptrim_duration":0.034210583000000003,"stat_sum":{"num_bytes":4963700736,"num_objects":2266,"num_object_clones":1292,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":276541,"num_read_kb":1085408,"num_write":272498,"num_write_kb":1089992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'2335","reported_seq":3761,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:29:51.526324+0000","last_change":"2023-09-23T08:18:50.540346+0000","last_active":"2023-09-23T08:29:51.526324+0000","last_peered":"2023-09-23T08:29:51.526324+0000","last_clean":"2023-09-23T08:29:51.526324+0000","last_became_active":"2023-09-23T07:12:04.898183+0000","last_became_peered":"2023-09-23T07:12:04.898183+0000","last_unstale":"2023-09-23T08:29:51.526324+0000","last_undegraded":"2023-09-23T08:29:51.526324+0000","last_fullsized":"2023-09-23T08:29:51.526324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2335","last_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_deep_scrub":"93'2335","last_deep_scrub_stamp":"2023-09-23T08:18:50.540192+0000","last_clean_scrub_stamp":"2023-09-23T08:18:50.540192+0000","objects_scrubbed":25,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:18:18.734077+0000","scrub_duration":0.35302462899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2032,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":14336,"num_read_kb":11946,"num_write":4469,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"102'292422","reported_seq":585564,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:03.084271+0000","last_change":"2023-09-23T08:25:53.021708+0000","last_active":"2023-09-23T08:30:03.084271+0000","last_peered":"2023-09-23T08:30:03.084271+0000","last_clean":"2023-09-23T08:30:03.084271+0000","last_became_active":"2023-09-23T07:12:14.742538+0000","last_became_peered":"2023-09-23T07:12:14.742538+0000","last_unstale":"2023-09-23T08:30:03.084271+0000","last_undegraded":"2023-09-23T08:30:03.084271+0000","last_fullsized":"2023-09-23T08:30:03.084271+0000","mapping_epoch":18,"log_start":"100'282419","ondisk_log_start":"100'282419","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'278971","last_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_deep_scrub":"99'278971","last_deep_scrub_stamp":"2023-09-23T08:25:53.021514+0000","last_clean_scrub_stamp":"2023-09-23T08:25:53.021514+0000","objects_scrubbed":2392,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":421,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:55:15.183609+0000","scrub_duration":420.50218997799999,"objects_trimmed":0,"snaptrim_duration":0.023452682999999998,"stat_sum":{"num_bytes":5292720128,"num_objects":2392,"num_object_clones":1357,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":294056,"num_read_kb":1154512,"num_write":291065,"num_write_kb":1164260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"102'4998","reported_seq":7665,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:02.760322+0000","last_change":"2023-09-23T08:18:47.399394+0000","last_active":"2023-09-23T08:30:02.760322+0000","last_peered":"2023-09-23T08:30:02.760322+0000","last_clean":"2023-09-23T08:30:02.760322+0000","last_became_active":"2023-09-23T07:12:04.900108+0000","last_became_peered":"2023-09-23T07:12:04.900108+0000","last_unstale":"2023-09-23T08:30:02.760322+0000","last_undegraded":"2023-09-23T08:30:02.760322+0000","last_fullsized":"2023-09-23T08:30:02.760322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4531","last_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_deep_scrub":"93'4531","last_deep_scrub_stamp":"2023-09-23T08:18:47.399181+0000","last_clean_scrub_stamp":"2023-09-23T08:18:47.399181+0000","objects_scrubbed":20,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:12:40.754199+0000","scrub_duration":0.23397675200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1088,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10559,"num_read_kb":8838,"num_write":2888,"num_write_kb":2833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2396,"num_omap_keys":42,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"102'5086","reported_seq":9613,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-09-23T08:30:01.766532+0000","last_change":"2023-09-23T08:18:58.397326+0000","last_active":"2023-09-23T08:30:01.766532+0000","last_peered":"2023-09-23T08:30:01.766532+0000","last_clean":"2023-09-23T08:30:01.766532+0000","last_became_active":"2023-09-23T07:12:04.897782+0000","last_became_peered":"2023-09-23T07:12:04.897782+0000","last_unstale":"2023-09-23T08:30:01.766532+0000","last_undegraded":"2023-09-23T08:30:01.766532+0000","last_fullsized":"2023-09-23T08:30:01.766532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'4474","last_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_deep_scrub":"93'4474","last_deep_scrub_stamp":"2023-09-23T08:18:58.397087+0000","last_clean_scrub_stamp":"2023-09-23T08:18:58.397087+0000","objects_scrubbed":15,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:04:59.467839+0000","scrub_duration":0.086441802999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3818,"num_read_kb":3291,"num_write":100,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2301,"num_omap_keys":74,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20426498067,"num_objects":9333,"num_object_clones":5332,"num_object_copies":27999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9333,"num_whiteouts":0,"num_read":1135266,"num_read_kb":4455808,"num_write":1120763,"num_write_kb":4483046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13431906304,"data_stored":54491598848,"data_compressed":36539902,"data_compressed_allocated":1684144128,"data_compressed_original":41377722368,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5714,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9257,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":80430,"num_read_kb":67673,"num_write":19561,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15179,"num_omap_keys":330,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":18514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44619,"internal_metadata":0},"log_size":38754,"ondisk_log_size":38754,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739303,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5904152,"kb_used_data":4372752,"kb_used_omap":573,"kb_used_meta":1530818,"kb_avail":88467688,"statfs":{"total":96636764160,"available":90590912512,"internally_reserved":0,"allocated":4477698048,"data_stored":18164258959,"data_compressed":11782859,"data_compressed_allocated":561381376,"data_compressed_original":13793312768,"omap_allocated":587472,"internal_metadata":1567557936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,10,5,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:29:51 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":2.8410000000000002,"15min":2.2909999999999999},"min":{"1min":0.20300000000000001,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":12.159000000000001,"5min":12.159000000000001,"15min":14.285},"last":3.5449999999999999},{"interface":"front","average":{"1min":2.6040000000000001,"5min":3.0270000000000001,"15min":2.3719999999999999},"min":{"1min":0.245,"5min":0.224,"15min":0.215},"max":{"1min":12.093999999999999,"5min":12.093999999999999,"15min":22.266999999999999},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Sep 23 08:29:51 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":3.242,"15min":3.1760000000000002},"min":{"1min":0.20499999999999999,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":12.327999999999999,"5min":22.099,"15min":22.099},"last":3.629},{"interface":"front","average":{"1min":2.5169999999999999,"5min":2.714,"15min":2.641},"min":{"1min":0.25700000000000001,"5min":0.22500000000000001,"15min":0.20599999999999999},"max":{"1min":12.4,"5min":22.135999999999999,"15min":28.257000000000001},"last":3.661}]}]},{"osd":0,"up_from":8,"seq":34359739300,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4980380,"kb_used_data":4373844,"kb_used_omap":582,"kb_used_meta":605945,"kb_avail":89391460,"statfs":{"total":96636764160,"available":91536855040,"internally_reserved":0,"allocated":4478816256,"data_stored":18164853724,"data_compressed":12371201,"data_compressed_allocated":561418240,"data_compressed_original":13792935936,"omap_allocated":596447,"internal_metadata":620488225},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,6,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 08:29:24 2023","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.1019999999999999,"15min":2.0990000000000002},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.182},"max":{"1min":7.492,"5min":15.090999999999999,"15min":15.090999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":3.7869999999999999,"5min":3.4830000000000001,"15min":3.2469999999999999},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.219},"max":{"1min":13.824,"5min":15.265000000000001,"15min":15.265000000000001},"last":1.6479999999999999}]},{"osd":2,"last update":"Sat Sep 23 08:29:24 2023","interfaces":[{"interface":"back","average":{"1min":3.4820000000000002,"5min":3.9609999999999999,"15min":3.4500000000000002},"min":{"1min":0.34200000000000003,"5min":0.215,"15min":0.19500000000000001},"max":{"1min":16.864999999999998,"5min":32.962000000000003,"15min":32.962000000000003},"last":9.6839999999999993},{"interface":"front","average":{"1min":2.2130000000000001,"5min":3.4289999999999998,"15min":2.9350000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":7.8959999999999999,"5min":32.088999999999999,"15min":32.088999999999999},"last":9.3390000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739299,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5899876,"kb_used_data":4373080,"kb_used_omap":583,"kb_used_meta":1526200,"kb_avail":88471964,"statfs":{"total":96636764160,"available":90595291136,"internally_reserved":0,"allocated":4478033920,"data_stored":18164878682,"data_compressed":12391066,"data_compressed_allocated":561418240,"data_compressed_original":13793943552,"omap_allocated":597990,"internal_metadata":1562828826},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,3,6,5,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 08:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":2.5800000000000001,"15min":2.5920000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":15.004,"5min":15.004,"15min":29.236999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":3.3959999999999999,"5min":3.2629999999999999,"15min":3.2149999999999999},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":9.5060000000000002,"5min":14.664999999999999,"15min":29.521000000000001},"last":8.6479999999999997}]},{"osd":2,"last update":"Sat Sep 23 08:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":3.2730000000000001,"15min":3.5470000000000002},"min":{"1min":0.27600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":12.513999999999999,"5min":14.301,"15min":20.216999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.3,"5min":1.4870000000000001,"15min":1.6679999999999999},"min":{"1min":0.23999999999999999,"5min":0.20100000000000001,"15min":0.193},"max":{"1min":7.1079999999999997,"5min":8.4329999999999998,"15min":18.353000000000002},"last":2.0870000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2857,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":4779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15462,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":9257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20013,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":4478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9144,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4477939712,"data_stored":18163863552,"data_compressed":12368304,"data_compressed_allocated":561381376,"data_compressed_original":13791961088,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4477095936,"data_stored":18163879936,"data_compressed":12388497,"data_compressed_allocated":561381376,"data_compressed_original":13792780288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4476870656,"data_stored":18163855360,"data_compressed":11783101,"data_compressed_allocated":561381376,"data_compressed_original":13792980992,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T08:30:05.763 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-23T08:30:06.445 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-23T08:30:06.446 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-23T08:30:06.446 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-23T08:30:06.446 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T08:30:08.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-09-23T08:30:12.548 INFO:tasks.ceph.osd.0:Stopped 2023-09-23T08:30:12.548 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-23T08:30:12.548 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T08:30:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-09-23T08:30:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-09-23T08:30:18.650 INFO:tasks.ceph.osd.1:Stopped 2023-09-23T08:30:18.650 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-23T08:30:18.650 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T08:30:18.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-23T08:30:18.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-09-23T08:30:24.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-09-23T08:30:24.752 INFO:tasks.ceph.osd.2:Stopped 2023-09-23T08:30:24.752 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-23T08:30:24.752 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-23T08:30:24.753 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T08:30:24.782 INFO:tasks.ceph.mgr.x:Stopped 2023-09-23T08:30:24.782 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-23T08:30:24.782 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-23T08:30:24.782 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T08:30:24.841 INFO:tasks.ceph.mon.a:Stopped 2023-09-23T08:30:24.841 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-23T08:30:24.842 DEBUG:teuthology.orchestra.run.smithi140:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 2023-09-23T08:30:24.982 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T08:30:24.983 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-23T08:30:26.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T08:30:26.503 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-23T08:30:26.773 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T08:30:26.773 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-23T08:30:27.031 INFO:tasks.ceph:Archiving mon data... 2023-09-23T08:30:27.033 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/data/mon.a.tgz 2023-09-23T08:30:27.033 DEBUG:teuthology.orchestra.run.smithi140:> mktemp 2023-09-23T08:30:27.071 INFO:teuthology.orchestra.run.smithi140.stdout:/tmp/tmp.t4Tuj1GPRy 2023-09-23T08:30:27.072 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.t4Tuj1GPRy 2023-09-23T08:30:27.251 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0666 /tmp/tmp.t4Tuj1GPRy 2023-09-23T08:30:27.389 DEBUG:teuthology.orchestra.remote:smithi140:/tmp/tmp.t4Tuj1GPRy is 471KB 2023-09-23T08:30:27.487 DEBUG:teuthology.orchestra.run.smithi140:> rm -fr /tmp/tmp.t4Tuj1GPRy 2023-09-23T08:30:27.527 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-23T08:30:27.527 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-23T08:30:27.635 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-23T08:30:27.635 DEBUG:teuthology.orchestra.run.smithi140:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-23T08:30:27.665 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-23T08:30:27.671 INFO:tasks.ceph:Archiving crash dumps... 2023-09-23T08:30:27.673 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/remote/smithi140/crash 2023-09-23T08:30:27.673 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-23T08:30:27.737 INFO:tasks.ceph:Compressing logs... 2023-09-23T08:30:27.738 DEBUG:teuthology.orchestra.run.smithi140:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-23T08:30:27.828 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122894.log 2023-09-23T08:30:27.828 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128933.log 2023-09-23T08:30:27.829 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118693.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121583.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128933.log: /var/log/ceph/ceph-client.admin.118693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131268.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124161.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122627.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131166.log 2023-09-23T08:30:27.830 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124133.log 2023-09-23T08:30:27.838 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130148.log 2023-09-23T08:30:27.838 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127755.log 2023-09-23T08:30:27.838 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134097.log 2023-09-23T08:30:27.838 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130741.log 2023-09-23T08:30:27.838 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134361.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130578.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131705.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118325.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131228.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127528.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118446.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133779.log 2023-09-23T08:30:27.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133417.log 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136471.log 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118693.log.gz -- replaced with /var/log/ceph/ceph-client.admin.118325.log.gz 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118446.log.gz 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130671.log 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121583.log.gz 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124161.log.gz 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124133.log.gz 2023-09-23T08:30:27.840 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122987.log 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136471.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127528.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127755.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128933.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129692.log 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122894.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122627.log.gz 2023-09-23T08:30:27.841 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123849.log 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122987.log.gz 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133147.log 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129692.log.gz 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127918.log 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123849.log.gz 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121217.log 2023-09-23T08:30:27.842 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130220.log 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127918.log.gz 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121796.log 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121217.log.gz 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129621.log 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129723.log 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121796.log.gz 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134298.log 2023-09-23T08:30:27.843 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129621.log.gz 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129723.log.gz 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137210.log 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114114.log 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134298.log: /var/log/ceph/ceph-client.admin.137210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134696.log 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137210.log.gz 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.114114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114114.log.gz 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118593.log 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131498.log 2023-09-23T08:30:27.844 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134696.log: /var/log/ceph/ceph-client.admin.118593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133644.log 2023-09-23T08:30:27.845 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118593.log.gz 2023-09-23T08:30:27.845 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131498.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.109351.log 2023-09-23T08:30:27.888 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122460.log 2023-09-23T08:30:27.888 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph.tmp-client.admin.109351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134647.log 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.109351.log.gz 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130148.log.gz 0.0% 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.131268.log.gz 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131166.log.gz 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130741.log.gz 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130220.log.gz 2023-09-23T08:30:27.889 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 /var/log/ceph/ceph-client.admin.122460.log: --verbose -- /var/log/ceph/ceph-client.admin.132656.log 0.0% 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.131228.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130578.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130671.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122460.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133644.log.gz -- replaced with /var/log/ceph/ceph-client.admin.133779.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133147.log.gz 2023-09-23T08:30:27.890 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133417.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134097.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134298.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134647.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134696.log.gz -- replaced with /var/log/ceph/ceph-client.admin.131498.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131705.log.gz -- replaced with /var/log/ceph/ceph-client.admin.134361.log.gz 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.891 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124811.log 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132656.log.gz 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128114.log 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127458.log 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124811.log.gz 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128114.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.128114.log.gz--verbose 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr:-- /var/log/ceph/ceph-client.admin.129859.log 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127293.log 2023-09-23T08:30:27.892 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127458.log.gz 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129859.log.gz 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127088.log 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137153.log 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127293.log.gz 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128044.log 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127088.log.gz 2023-09-23T08:30:27.893 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.137153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127848.log 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137153.log.gz 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134554.log 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128044.log.gz 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123775.log 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127848.log.gz 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125472.log 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134554.log.gz 2023-09-23T08:30:27.894 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123775.log.gz 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120316.log 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128610.log 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125472.log.gz 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128312.log 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120316.log.gz 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131434.log 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128610.log.gz 2023-09-23T08:30:27.895 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126763.log 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128312.log.gz 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128808.log 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131434.log.gz 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131931.log 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126763.log.gz 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129496.log 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128808.log.gz 2023-09-23T08:30:27.896 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131931.log.gz 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133613.log 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117809.log 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129496.log.gz 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133613.log.gz 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131806.log 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135553.log 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117809.log.gz 2023-09-23T08:30:27.897 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121143.log 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131806.log.gz 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135553.log.gz 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131395.log 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122116.log 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121143.log.gz 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129590.log 2023-09-23T08:30:27.898 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131395.log.gz 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123933.log 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122116.log.gz 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121999.log 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129590.log.gz 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132428.log 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123933.log.gz 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122925.log 2023-09-23T08:30:27.899 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121999.log.gz 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132859.log 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132428.log.gz 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126734.log 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122925.log.gz 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132859.log.gz 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123640.log 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133975.log 2023-09-23T08:30:27.900 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126734.log.gz 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123032.log 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123640.log.gz 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122218.log 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133975.log.gz 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128509.log 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123032.log.gz 2023-09-23T08:30:27.901 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123820.log 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122218.log.gz 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132333.log 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128509.log.gz 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118204.log 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123820.log.gz 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124231.log 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132333.log.gz 2023-09-23T08:30:27.902 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127150.log 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118204.log.gz 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129559.log 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124231.log.gz 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123587.log 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127150.log.gz 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129559.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.129559.log.gz/var/log/ceph/ceph-client.admin.126158.log 2023-09-23T08:30:27.903 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123587.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123587.log.gz 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130772.log 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126158.log.gz 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124293.log 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130772.log.gz 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124657.log 2023-09-23T08:30:27.904 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131300.log 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124293.log.gz 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136522.log 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124657.log.gz 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134316.log 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131300.log.gz 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123122.log 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136522.log.gz 2023-09-23T08:30:27.905 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133014.log 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134316.log.gz 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123122.log.gz 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134141.log 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127365.log 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133014.log.gz 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136632.log 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134141.log.gz 2023-09-23T08:30:27.906 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124718.log 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127365.log.gz 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136632.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.120927.log 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.136632.log.gz 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124718.log.gz 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124895.log 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136897.log 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120927.log.gz 2023-09-23T08:30:27.907 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121854.log 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124895.log.gz 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134242.log 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136897.log.gz 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121854.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.126649.log -- replaced with /var/log/ceph/ceph-client.admin.121854.log.gz 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134242.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.134893.log 2023-09-23T08:30:27.908 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.134242.log.gz 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125500.log 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126649.log.gz 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125417.log 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134893.log.gz 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118740.log 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125500.log.gz 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136740.log 2023-09-23T08:30:27.909 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125417.log.gz 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118740.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.118740.log.gz 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:--verbose -- /var/log/ceph/ceph-client.admin.124034.log 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136740.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.136740.log.gz/var/log/ceph/ceph-client.admin.121234.log 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124034.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.124034.log.gz-- 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133386.log 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120370.log 2023-09-23T08:30:27.910 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121234.log.gz 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131134.log 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133386.log.gz 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136848.log 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120370.log.gz 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124262.log 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131134.log.gz 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135187.log 2023-09-23T08:30:27.911 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136848.log.gz 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124262.log.gz 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125008.log 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125799.log 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135187.log.gz 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125008.log.gz 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130117.log 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123736.log 2023-09-23T08:30:27.912 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125799.log.gz 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130117.log.gz 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121271.log 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134536.log 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123736.log.gz 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120569.log 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121271.log.gz 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137308.log 2023-09-23T08:30:27.913 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134536.log.gz 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129394.log 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120569.log.gz 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.137308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129066.log 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137308.log.gz 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128446.log 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129394.log.gz 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130837.log 2023-09-23T08:30:27.914 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129066.log.gz 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128414.log 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128446.log.gz 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136421.log 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130837.log.gz 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128414.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.128414.log.gz/var/log/ceph/ceph-client.admin.133249.log 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:27.915 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121197.log 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136421.log.gz 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124003.log 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133249.log.gz 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125156.log 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121197.log.gz 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124003.log.gzgzip 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.126094.log 2023-09-23T08:30:27.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130710.log 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125156.log.gz 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126094.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126094.log.gz 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132983.log 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130710.log.gz 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136995.log 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121912.log 2023-09-23T08:30:27.917 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132983.log.gz 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122057.log 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136995.log.gz 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130086.log 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121912.log.gz 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120694.log 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122057.log.gz 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125634.log 2023-09-23T08:30:27.918 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130086.log.gz 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117711.log 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120694.log.gz 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124192.log 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125634.log.gz 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132197.log 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117711.log.gz 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131072.log 2023-09-23T08:30:27.919 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124192.log.gz 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132593.log 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132197.log.gz 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132921.log 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131072.log.gz 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132593.log.gz 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128013.log 2023-09-23T08:30:27.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132921.log.gz 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123964.log 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133053.log 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128013.log.gz 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133709.log 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123964.log.gz 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121508.log 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133053.log.gz 2023-09-23T08:30:27.921 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130477.log 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133709.log.gz 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134482.log 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121508.log.gz 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121344.log 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130477.log.gz 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125996.log 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134482.log.gz 2023-09-23T08:30:27.922 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125833.log 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121344.log.gz 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126586.log 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125996.log.gz 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129891.log 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125833.log.gz 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126527.log 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126586.log.gz 2023-09-23T08:30:27.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128375.log 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129891.log.gz 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117613.log 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126527.log.gz 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120549.log 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128375.log.gz 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133841.log 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117613.log.gz 2023-09-23T08:30:27.924 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136789.log 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120549.log.gz 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130907.log 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133841.log.gz 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136789.log.gz 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126389.log 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120947.log 2023-09-23T08:30:27.925 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130907.log.gz 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120353.log 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126389.log.gz 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120839.log 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120947.log.gz 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120353.log.gz 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135346.log 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122728.log 2023-09-23T08:30:27.926 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120839.log.gz 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120785.log 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135346.log.gz 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135294.log 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122728.log.gz 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121109.log 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120785.log.gz 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117544.log 2023-09-23T08:30:27.927 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135294.log.gz 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128478.log 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121109.log.gz 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129332.log 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117544.log.gz 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132952.log 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128478.log.gz 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133872.log 2023-09-23T08:30:27.928 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129332.log.gz 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125967.log 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132952.log.gz 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135856.log 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133872.log.gz 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132625.log 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125967.log.gz 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130384.log 2023-09-23T08:30:27.929 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135856.log.gz 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126024.log 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132625.log.gz 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120640.log 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130384.log.gz 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135757.log 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126024.log.gz 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126203.log 2023-09-23T08:30:27.930 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120640.log.gz 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130508.log 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135757.log.gz 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124749.log 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126203.log.gz 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120387.log 0.0% 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.130508.log.gz 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128902.log 2023-09-23T08:30:27.931 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124749.log.gz 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122666.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120387.log.gz 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120910.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126232.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134068.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121035.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128972.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126502.log 2023-09-23T08:30:27.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132890.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126788.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134437.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130971.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136683.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121288.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127034.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-09-23T08:30:27.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135089.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136364.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133085.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132096.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127591.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126994.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132820.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123443.log 2023-09-23T08:30:27.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135455.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120802.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129136.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120259.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123474.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134991.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126358.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124075.log 2023-09-23T08:30:27.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121072.log 2023-09-23T08:30:27.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132695.log 2023-09-23T08:30:27.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127949.log 2023-09-23T08:30:27.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118351.log 2023-09-23T08:30:27.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128280.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127724.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137054.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122514.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124423.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123874.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121251.log 2023-09-23T08:30:28.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121680.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126063.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128145.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134518.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131838.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120603.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127427.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118005.log 2023-09-23T08:30:28.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118057.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131332.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121325.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127559.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132789.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125733.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121018.log 2023-09-23T08:30:28.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127489.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134580.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134745.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135040.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133314.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127059.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122405.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131562.log 2023-09-23T08:30:28.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121470.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122827.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125301.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132561.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129984.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125928.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134334.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120441.log 2023-09-23T08:30:28.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130313.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120586.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129922.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136209.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128870.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133347.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130547.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134159.log 2023-09-23T08:30:28.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126949.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124490.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131103.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131601.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128704.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133448.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122376.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117453.log 2023-09-23T08:30:28.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125210.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128083.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124369.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136160.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122697.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137260.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118642.log 2023-09-23T08:30:28.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120276.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132166.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131971.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123086.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132460.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133479.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132261.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129660.log 2023-09-23T08:30:28.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125039.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121621.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124104.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124842.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126618.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120296.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120748.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135504.log 2023-09-23T08:30:28.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134456.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135651.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127685.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126293.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122802.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135807.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120461.log 2023-09-23T08:30:28.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134794.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121546.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128641.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133178.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118544.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117908.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123505.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125679.log 2023-09-23T08:30:28.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120407.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127817.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120765.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124324.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130805.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132127.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121394.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125446.log 2023-09-23T08:30:28.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117760.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129230.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131900.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129828.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125372.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134843.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135914.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134399.log 2023-09-23T08:30:28.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120478.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120515.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132033.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130189.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131002.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123704.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120424.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129457.log 2023-09-23T08:30:28.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133810.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126555.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123381.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125603.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123246.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129757.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117662.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122568.log 2023-09-23T08:30:28.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124780.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130282.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121180.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123291.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122855.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132530.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118495.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129426.log 2023-09-23T08:30:28.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126878.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136266.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133217.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121126.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121055.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124552.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126473.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123558.log 2023-09-23T08:30:28.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125238.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135397.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121305.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120532.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130345.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125110.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129198.log 2023-09-23T08:30:28.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131364.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123153.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130940.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126816.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134942.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126847.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127653.log 2023-09-23T08:30:28.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127786.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131663.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128839.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125532.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120495.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131033.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128774.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128177.log 2023-09-23T08:30:28.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124521.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125895.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130446.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117910.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127396.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134216.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133543.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129789.log 2023-09-23T08:30:28.021 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121950.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118152.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125864.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123345.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136579.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130609.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123902.log 2023-09-23T08:30:28.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129262.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134260.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120981.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131869.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127190.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124924.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126689.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127622.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136111.log 2023-09-23T08:30:28.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123061.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128735.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134500.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120819.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122543.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121001.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132228.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132064.log 2023-09-23T08:30:28.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118178.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125762.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126257.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125571.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120711.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124459.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128344.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134006.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122172.log 2023-09-23T08:30:28.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134280.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126420.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123612.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120856.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129167.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120893.log 2023-09-23T08:30:28.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122773.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130251.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130415.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132727.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132364.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120333.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114218.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131632.log 2023-09-23T08:30:28.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129097.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124398.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135138.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127119.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124628.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130055.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127221.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129363.log 2023-09-23T08:30:28.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120964.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123320.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123671.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128673.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132499.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133903.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131197.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130868.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121089.log 2023-09-23T08:30:28.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126918.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120873.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114170.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120731.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129953.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132395.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133943.log 2023-09-23T08:30:28.030 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128209.log 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131736.log 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121737.log 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128902.log: /var/log/ceph/ceph-client.admin.122666.log: /var/log/ceph/ceph-client.admin.120910.log: /var/log/ceph/ceph-client.admin.126232.log: /var/log/ceph/ceph-client.admin.131736.log: /var/log/ceph/ceph-client.admin.134068.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128902.log.gz -- replaced with /var/log/ceph/ceph-client.admin.122666.log.gz 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120910.log.gz 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126232.log.gz 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131736.log.gz 2023-09-23T08:30:28.031 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134068.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125270.log 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121737.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121035.log: /var/log/ceph/ceph-client.admin.128972.log: /var/log/ceph/ceph-client.admin.126502.log: /var/log/ceph/ceph-client.admin.132890.log: /var/log/ceph/ceph-client.admin.126788.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121035.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128972.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126502.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132890.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126788.log.gz 2023-09-23T08:30:28.032 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134437.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130971.log: /var/log/ceph/ceph-client.admin.136683.log: /var/log/ceph/ceph-client.admin.125270.log: /var/log/ceph/ceph-client.admin.121288.log: /var/log/ceph/ceph-client.admin.127034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130971.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-osd.0.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136683.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125270.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121288.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127034.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120242.log 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135089.log: /var/log/ceph/ceph-client.admin.136364.log: /var/log/ceph/ceph-client.admin.133085.log: /var/log/ceph/ceph-client.admin.132096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135089.log.gz 2023-09-23T08:30:28.033 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136364.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133085.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132096.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126994.log: 0.0%/var/log/ceph/ceph-client.admin.132820.log: -- replaced with /var/log/ceph/ceph-client.admin.127591.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124690.log 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126994.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132820.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123443.log: /var/log/ceph/ceph-client.admin.135455.log: /var/log/ceph/ceph-client.admin.120802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123443.log.gz 2023-09-23T08:30:28.034 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135455.log.gz 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120802.log.gz 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120242.log.gz 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120259.log: /var/log/ceph/ceph-client.admin.123474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129136.log.gzgzip 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.127982.log 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120259.log.gz 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123474.log.gz 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134991.log: /var/log/ceph/ceph-client.admin.126358.log: /var/log/ceph/ceph-client.admin.124690.log: /var/log/ceph/ceph-client.admin.124075.log: /var/log/ceph/ceph-client.admin.121072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134991.log.gz 0.0% 2023-09-23T08:30:28.035 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126358.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124690.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124075.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121072.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132695.log: /var/log/ceph/ceph-client.admin.127949.log: /var/log/ceph/ceph-client.admin.118351.log: /var/log/ceph/ceph-client.admin.128280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132695.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127724.log: /var/log/ceph/ceph-client.admin.127982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127949.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118351.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128280.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127724.log.gz 2023-09-23T08:30:28.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.137054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127982.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137054.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125185.log 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122514.log: /var/log/ceph/ceph-client.admin.124423.log: /var/log/ceph/ceph-client.admin.123874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122514.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124423.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123874.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121251.log: /var/log/ceph/ceph-client.admin.121680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121251.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121680.log.gz 2023-09-23T08:30:28.037 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133116.log 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126063.log: /var/log/ceph/ceph-client.admin.128145.log: /var/log/ceph/ceph-client.admin.134518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126063.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128145.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134518.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125185.log: /var/log/ceph/ceph-client.admin.131838.log: /var/log/ceph/ceph-client.admin.120603.log: /var/log/ceph/ceph-client.admin.127427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125185.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131838.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120603.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118005.log.gz 2023-09-23T08:30:28.038 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127427.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118057.log: /var/log/ceph/ceph-client.admin.131332.log: /var/log/ceph/ceph-client.admin.121325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125708.log 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118057.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131332.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121325.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133116.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127559.log: /var/log/ceph/ceph-client.admin.132789.log: /var/log/ceph/ceph-client.admin.125733.log: /var/log/ceph/ceph-client.admin.121018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127559.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132789.log.gz 2023-09-23T08:30:28.039 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125733.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121018.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132002.log 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127489.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134580.log: /var/log/ceph/ceph-client.admin.134745.log: /var/log/ceph/ceph-client.admin.125708.log: /var/log/ceph/ceph-client.admin.135040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134580.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134745.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125708.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135040.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133314.log: /var/log/ceph/ceph-client.admin.127059.log: /var/log/ceph/ceph-client.admin.122405.log: /var/log/ceph/ceph-client.admin.132002.log: /var/log/ceph/ceph-client.admin.131562.log: /var/log/ceph/ceph-client.admin.121470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133314.log.gz 2023-09-23T08:30:28.040 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122405.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127059.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132002.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131562.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121470.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123215.log 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122827.log: /var/log/ceph/ceph-client.admin.125301.log: /var/log/ceph/ceph-client.admin.132561.log: 0.0%/var/log/ceph/ceph-client.admin.129984.log: -- replaced with /var/log/ceph/ceph-client.admin.122827.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125301.log.gz 2023-09-23T08:30:28.041 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132561.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129984.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125928.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134334.log: /var/log/ceph/ceph-client.admin.120441.log: /var/log/ceph/ceph-client.admin.130313.log: 0.0%/var/log/ceph/ceph-client.admin.120586.log: -- replaced with /var/log/ceph/ceph-client.admin.134334.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120441.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130313.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120586.log.gz 2023-09-23T08:30:28.042 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129922.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120623.log 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136209.log: /var/log/ceph/ceph-client.admin.128870.log: /var/log/ceph/ceph-client.admin.133347.log: /var/log/ceph/ceph-client.admin.123215.log: /var/log/ceph/ceph-client.admin.130547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136209.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128870.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133347.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123215.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130547.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134159.log: /var/log/ceph/ceph-client.admin.126949.log: /var/log/ceph/ceph-client.admin.124490.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134159.log.gz -- replaced with /var/log/ceph/ceph-client.admin.126949.log.gz 2023-09-23T08:30:28.043 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124490.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134197.log 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131103.log: /var/log/ceph/ceph-client.admin.131601.log: /var/log/ceph/ceph-client.admin.128704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131103.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131601.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128704.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120623.log.gz/var/log/ceph/ceph-client.admin.133448.log: 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122376.log: /var/log/ceph/ceph-client.admin.117453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133448.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122376.log.gz 2023-09-23T08:30:28.044 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117453.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125210.log: /var/log/ceph/ceph-client.admin.128083.log: /var/log/ceph/ceph-client.admin.134197.log: /var/log/ceph/ceph-client.admin.124369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134197.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125210.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128083.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124369.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128249.log 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136160.log: /var/log/ceph/ceph-client.admin.122697.log: /var/log/ceph/ceph-client.admin.137260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136160.log.gz/var/log/ceph/ceph-client.admin.118642.log: 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122697.log.gz 2023-09-23T08:30:28.045 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137260.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118642.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128571.log 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120276.log: /var/log/ceph/ceph-client.admin.132166.log: /var/log/ceph/ceph-client.admin.131971.log: /var/log/ceph/ceph-client.admin.123086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120276.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132166.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131971.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123086.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128249.log: /var/log/ceph/ceph-client.admin.132460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128249.log.gz 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114343.log 2023-09-23T08:30:28.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133479.log: /var/log/ceph/ceph-client.admin.132261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132460.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133479.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132261.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125070.log 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128571.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129660.log: /var/log/ceph/ceph-client.admin.125039.log: /var/log/ceph/ceph-client.admin.121621.log: /var/log/ceph/ceph-client.admin.124104.log: /var/log/ceph/ceph-client.admin.114343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129660.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125039.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121621.log.gz 2023-09-23T08:30:28.047 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124104.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114343.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124842.log: /var/log/ceph/ceph-client.admin.126618.log: /var/log/ceph/ceph-client.admin.120296.log: /var/log/ceph/ceph-client.admin.120748.log: /var/log/ceph/ceph-client.admin.135504.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126618.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.124842.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120296.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120748.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135504.log.gz 2023-09-23T08:30:28.048 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125070.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136946.log 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134456.log: /var/log/ceph/ceph-client.admin.135651.log: /var/log/ceph/ceph-client.admin.127685.log: /var/log/ceph/ceph-client.admin.126293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134456.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0%/var/log/ceph/ceph-client.admin.122802.log: -- replaced with /var/log/ceph/ceph-client.admin.135651.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127685.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126293.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122802.log.gz 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131531.log 2023-09-23T08:30:28.049 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135807.log: /var/log/ceph/ceph-client.admin.120461.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135807.log.gz 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120461.log.gz 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134794.log: /var/log/ceph/ceph-client.admin.136946.log: /var/log/ceph/ceph-client.admin.121546.log: /var/log/ceph/ceph-client.admin.128641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136946.log.gz 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134794.log.gz 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121546.log.gz 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133511.log 2023-09-23T08:30:28.050 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128641.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133178.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118544.log: /var/log/ceph/ceph-client.admin.117908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134123.log /var/log/ceph/ceph-client.admin.123505.log: 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131531.log: /var/log/ceph/ceph-client.admin.125679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118544.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117908.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123505.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131531.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125679.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120407.log: /var/log/ceph/ceph-client.admin.127817.log: /var/log/ceph/ceph-client.admin.133511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120407.log.gz 2023-09-23T08:30:28.051 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127817.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133511.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120765.log: /var/log/ceph/ceph-client.admin.124324.log: /var/log/ceph/ceph-client.admin.130805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120765.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124324.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130805.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134123.log: /var/log/ceph/ceph-client.admin.132127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134123.log.gz 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121394.log: /var/log/ceph/ceph-client.admin.125446.log: /var/log/ceph/ceph-client.admin.117760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132127.log.gzgzip 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr:-5 /var/log/ceph/ceph-client.admin.129230.log: --verbose -- /var/log/ceph/ceph-client.admin.136013.log 2023-09-23T08:30:28.052 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121394.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125446.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117760.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129230.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131900.log: /var/log/ceph/ceph-client.admin.129828.log: /var/log/ceph/ceph-client.admin.125372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131900.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129828.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125372.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134843.log: /var/log/ceph/ceph-client.admin.135914.log: /var/log/ceph/ceph-client.admin.134399.log: /var/log/ceph/ceph-client.admin.120478.log: /var/log/ceph/ceph-client.admin.120515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134843.log.gz 2023-09-23T08:30:28.053 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135914.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134399.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120478.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120515.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132033.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134037.log 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136013.log: /var/log/ceph/ceph-client.admin.130189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136013.log.gz 2023-09-23T08:30:28.054 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131002.log: /var/log/ceph/ceph-client.admin.123704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130189.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131002.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123704.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120424.log: /var/log/ceph/ceph-client.admin.129457.log: /var/log/ceph/ceph-client.admin.133810.log: /var/log/ceph/ceph-client.admin.126555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120424.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129457.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133810.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126555.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123381.log: /var/log/ceph/ceph-client.admin.134037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123381.log.gz 2023-09-23T08:30:28.055 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134037.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123184.log 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123246.log: /var/log/ceph/ceph-client.admin.129757.log: /var/log/ceph/ceph-client.admin.117662.log: 0.0%/var/log/ceph/ceph-client.admin.122568.log: -- replaced with /var/log/ceph/ceph-client.admin.125603.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123246.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129757.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117662.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122568.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124780.log: /var/log/ceph/ceph-client.admin.130282.log: /var/log/ceph/ceph-client.admin.121180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124780.log.gz 2023-09-23T08:30:28.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130282.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121180.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128540.log 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123291.log: /var/log/ceph/ceph-client.admin.122855.log: /var/log/ceph/ceph-client.admin.132530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123291.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122855.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132530.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123184.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118495.log: /var/log/ceph/ceph-client.admin.129426.log: /var/log/ceph/ceph-client.admin.126878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118495.log.gz 2023-09-23T08:30:28.057 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129426.log.gz 0.0% 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126878.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136266.log: /var/log/ceph/ceph-client.admin.133217.log: /var/log/ceph/ceph-client.admin.121126.log: /var/log/ceph/ceph-client.admin.121055.log: /var/log/ceph/ceph-client.admin.128540.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136266.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.133217.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121126.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121055.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128540.log.gz 2023-09-23T08:30:28.058 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124552.log: /var/log/ceph/ceph-client.admin.126473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122596.log 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124552.log.gz 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126473.log.gz 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123558.log: /var/log/ceph/ceph-client.admin.125238.log: /var/log/ceph/ceph-client.admin.135397.log: /var/log/ceph/ceph-client.admin.121305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123558.log.gz 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125238.log.gz 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135397.log.gz -- replaced with /var/log/ceph/ceph-client.admin.121305.log.gz 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120532.log: /var/log/ceph/ceph-client.admin.130345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129034.log 2023-09-23T08:30:28.059 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120532.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130345.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125110.log: /var/log/ceph/ceph-client.admin.129198.log: /var/log/ceph/ceph-client.admin.131364.log: /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.122596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129198.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125110.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131364.log.gzgzip 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.133281.log 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122596.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123153.log: /var/log/ceph/ceph-client.admin.130940.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130940.log.gz 2023-09-23T08:30:28.060 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.123153.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129034.log: /var/log/ceph/ceph-client.admin.126816.log: /var/log/ceph/ceph-client.admin.134942.log: /var/log/ceph/ceph-client.admin.126847.log: /var/log/ceph/ceph-client.admin.127653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126816.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129034.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134942.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126847.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127653.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127786.log: /var/log/ceph/ceph-client.admin.131663.log: /var/log/ceph/ceph-client.admin.128839.log: /var/log/ceph/ceph-client.admin.125532.log: /var/log/ceph/ceph-client.admin.120495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127786.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131663.log.gz 2023-09-23T08:30:28.061 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128839.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125532.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120495.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131033.log: /var/log/ceph/ceph-client.admin.128774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131033.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.128177.log: /var/log/ceph/ceph-client.admin.133281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128774.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128177.log.gz 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136315.log 2023-09-23T08:30:28.062 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124521.log: /var/log/ceph/ceph-client.admin.125895.log: 0.0%/var/log/ceph/ceph-client.admin.130446.log: -- replaced with /var/log/ceph/ceph-client.admin.133281.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124521.log.gz/var/log/ceph/ceph-client.admin.127396.log: 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125895.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130446.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117910.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127396.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134216.log: /var/log/ceph/ceph-client.admin.133543.log: /var/log/ceph/ceph-client.admin.129789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134216.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133543.log.gz 2023-09-23T08:30:28.063 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129789.log.gz 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127887.log 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121950.log.gz 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136315.log: /var/log/ceph/ceph-client.admin.118152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137104.log 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136315.log.gz 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118152.log.gz 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125864.log.gz 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123345.log: /var/log/ceph/ceph-client.admin.127887.log: /var/log/ceph/ceph-client.admin.136579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130640.log 2023-09-23T08:30:28.064 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123345.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127887.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136579.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130609.log: /var/log/ceph/ceph-client.admin.123902.log: /var/log/ceph/ceph-client.admin.137104.log: /var/log/ceph/ceph-client.admin.129262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130609.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123902.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137104.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129262.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134260.log: /var/log/ceph/ceph-client.admin.120981.log: /var/log/ceph/ceph-client.admin.131869.log: /var/log/ceph/ceph-client.admin.127190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120981.log.gz 2023-09-23T08:30:28.065 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134260.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131869.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127190.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130640.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126689.log: /var/log/ceph/ceph-client.admin.127622.log: /var/log/ceph/ceph-client.admin.136111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130023.log 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124924.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126689.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136111.log.gz 2023-09-23T08:30:28.066 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127622.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123061.log: /var/log/ceph/ceph-client.admin.128735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128735.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124583.log 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134500.log: /var/log/ceph/ceph-client.admin.120819.log: 0.0%/var/log/ceph/ceph-client.admin.122543.log: -- replaced with /var/log/ceph/ceph-client.admin.134500.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122543.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.120819.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130023.log: /var/log/ceph/ceph-client.admin.121001.log: /var/log/ceph/ceph-client.admin.132228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130023.log.gz 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.118299.log 2023-09-23T08:30:28.067 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121001.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132228.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132064.log: /var/log/ceph/ceph-client.admin.118178.log: /var/log/ceph/ceph-client.admin.125762.log: /var/log/ceph/ceph-client.admin.126257.log: /var/log/ceph/ceph-client.admin.124583.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132064.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118178.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125762.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126257.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124583.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125571.log: /var/log/ceph/ceph-client.admin.120711.log: /var/log/ceph/ceph-client.admin.124459.log: /var/log/ceph/ceph-client.admin.128344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125571.log.gz 2023-09-23T08:30:28.068 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120711.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124459.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128344.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134006.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118299.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122430.log 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.134280.log: /var/log/ceph/ceph-client.admin.126420.log: /var/log/ceph/ceph-client.admin.123612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122172.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134280.log.gz 2023-09-23T08:30:28.069 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126420.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123612.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120856.log: /var/log/ceph/ceph-client.admin.129167.log: /var/log/ceph/ceph-client.admin.120893.log: /var/log/ceph/ceph-client.admin.122773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120856.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129167.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120893.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122773.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.130251.log: /var/log/ceph/ceph-client.admin.130415.log: /var/log/ceph/ceph-client.admin.122430.log: /var/log/ceph/ceph-client.admin.132727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130251.log.gz 2023-09-23T08:30:28.070 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130415.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122430.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132727.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121163.log 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132364.log: /var/log/ceph/ceph-client.admin.120333.log: /var/log/ceph/ceph-client.admin.114218.log: /var/log/ceph/ceph-client.admin.131632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132364.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120333.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114218.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131632.log.gz 2023-09-23T08:30:28.071 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.129097.log: -- /var/log/ceph/ceph-client.admin.120677.log 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129097.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124398.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135138.log: /var/log/ceph/ceph-client.admin.121163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135138.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127119.log: /var/log/ceph/ceph-client.admin.124628.log: 0.0%/var/log/ceph/ceph-client.admin.130055.log: -- replaced with /var/log/ceph/ceph-client.admin.121163.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127119.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124628.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130055.log.gz 2023-09-23T08:30:28.072 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113930.log 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127221.log: /var/log/ceph/ceph-client.admin.129363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127221.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0%/var/log/ceph/ceph-client.admin.120964.log: -- replaced with /var/log/ceph/ceph-client.admin.129363.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120964.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120677.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134417.log 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123320.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123671.log.gz 2023-09-23T08:30:28.073 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.113930.log: /var/log/ceph/ceph-client.admin.128673.log: /var/log/ceph/ceph-client.admin.132499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113930.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128673.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132499.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133903.log: /var/log/ceph/ceph-client.admin.134417.log: /var/log/ceph/ceph-client.admin.131197.log: /var/log/ceph/ceph-client.admin.130868.log: 0.0%/var/log/ceph/ceph-client.admin.121089.log: -- replaced with /var/log/ceph/ceph-client.admin.133903.log.gz 0.0% 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.134417.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131197.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130868.log.gz 2023-09-23T08:30:28.074 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120225.log 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121089.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126918.log: /var/log/ceph/ceph-client.admin.120873.log: /var/log/ceph/ceph-client.admin.114170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126918.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120873.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114170.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120731.log: /var/log/ceph/ceph-client.admin.129953.log: /var/log/ceph/ceph-client.admin.132395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120731.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129953.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132395.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120225.log.gz 2023-09-23T08:30:28.075 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131767.log 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135602.log 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134178.log 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131767.log.gz 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135602.log.gz 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123412.log 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134178.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134178.log.gz 2023-09-23T08:30:28.076 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.123412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124949.log 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123412.log.gz 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133943.log: /var/log/ceph/ceph-client.admin.128209.log: /var/log/ceph/ceph-osd.1.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133943.log.gz 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128209.log.gz 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127325.log 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124949.log.gz 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122956.log 2023-09-23T08:30:28.077 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126325.log 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127325.log.gz 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126126.log 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122956.log.gz 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129301.log 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126325.log.gz 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.126126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122269.log 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126126.log.gz 2023-09-23T08:30:28.078 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129301.log.gzgzip 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.131465.log 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.122269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135963.log 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122269.log.gz 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.131465.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.131465.log.gz-5 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:--verbose -- /var/log/ceph/ceph-client.admin.133582.log 2023-09-23T08:30:28.079 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135963.log.gz 2023-09-23T08:30:28.081 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132758.log 2023-09-23T08:30:28.081 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135237.log 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133582.log.gz 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132758.log.gz 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125340.log 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135700.log 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135237.log.gz 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.125340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134381.log 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125340.log.gz 2023-09-23T08:30:28.082 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.135700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118056.log 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135700.log.gz 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.134381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129003.log 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134381.log.gz 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.118056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118056.log.gz 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124977.log 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136062.log 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129003.log.gz 2023-09-23T08:30:28.083 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.124977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120657.log 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124977.log.gz 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114411.log 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.136062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136062.log.gz 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.120657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129528.log 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120657.log.gz 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.114411.log: gzip -5 0.0%--verbose -- /var/log/ceph/ceph-client.admin.117909.log 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114411.log.gz 2023-09-23T08:30:28.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.129528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132293.log 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129528.log.gz 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133748.log 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.117909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117909.log.gz 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.132293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132293.log.gz 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127262.log 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133678.log 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133748.log.gz 2023-09-23T08:30:28.085 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.127262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127262.log.gz 2023-09-23T08:30:28.086 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121431.log 2023-09-23T08:30:28.086 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.133678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133678.log.gz 2023-09-23T08:30:28.086 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph-client.admin.121431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121431.log.gz 2023-09-23T08:30:28.086 INFO:teuthology.orchestra.run.smithi140.stderr: 25.8% -- replaced with /var/log/ceph/ceph-client.admin.123061.log.gz 2023-09-23T08:30:28.197 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-09-23T08:30:29.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-23T08:30:32.373 INFO:teuthology.orchestra.run.smithi140.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-09-23T08:30:37.772 INFO:teuthology.orchestra.run.smithi140.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-09-23T08:40:10.293 INFO:teuthology.orchestra.run.smithi140.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-09-23T08:47:40.569 INFO:teuthology.orchestra.run.smithi140.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-09-23T08:47:41.746 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-09-23T08:47:41.748 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-09-23T08:47:41.748 INFO:teuthology.orchestra.run.smithi140.stderr:real 17m13.952s 2023-09-23T08:47:41.748 INFO:teuthology.orchestra.run.smithi140.stderr:user 19m32.419s 2023-09-23T08:47:41.749 INFO:teuthology.orchestra.run.smithi140.stderr:sys 0m50.631s 2023-09-23T08:47:41.749 INFO:tasks.ceph:Archiving logs... 2023-09-23T08:47:41.751 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/log/ceph to /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/remote/smithi140/log 2023-09-23T08:47:41.752 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-23T08:50:35.449 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-23T08:50:35.477 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-23T08:50:35.477 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-23T08:50:35.607 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-23T08:50:35.616 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-23T08:50:35.617 DEBUG:teuthology.orchestra.run.smithi140:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-23T08:50:35.636 INFO:teuthology.orchestra.run.smithi140.stderr:bash: ntpq: command not found 2023-09-23T08:50:35.737 INFO:teuthology.orchestra.run.smithi140.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-23T08:50:35.737 INFO:teuthology.orchestra.run.smithi140.stdout:=============================================================================== 2023-09-23T08:50:35.738 INFO:teuthology.orchestra.run.smithi140.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 80 -445us[ -445us] +/- 86ms 2023-09-23T08:50:35.738 INFO:teuthology.orchestra.run.smithi140.stdout:^* hv02.front.sepia.ceph.com 3 9 377 344 -1217us[-1191us] +/- 65ms 2023-09-23T08:50:35.738 INFO:teuthology.orchestra.run.smithi140.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 19 +2768us[+2768us] +/- 63ms 2023-09-23T08:50:35.738 INFO:teuthology.orchestra.run.smithi140.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-09-23T08:50:35.739 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-23T08:50:35.748 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-23T08:50:35.748 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-23T08:50:35.768 DEBUG:teuthology.orchestra.run.smithi140:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-23T08:50:36.028 DEBUG:teuthology.orchestra.run.smithi140:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-23T08:50:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T08:50:36.062 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-23T08:50:36.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-23T08:50:36.109 INFO:teuthology.task.internal:Duration was 6604.764305 seconds 2023-09-23T08:50:36.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-23T08:50:36.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-23T08:50:36.143 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-23T08:50:36.262 INFO:teuthology.orchestra.run.smithi140.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-23T08:50:36.616 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-23T08:50:36.617 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi140.front.sepia.ceph.com 2023-09-23T08:50:36.617 DEBUG:teuthology.orchestra.run.smithi140:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-23T08:50:36.696 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-23T08:50:36.697 DEBUG:teuthology.orchestra.run.smithi140:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-23T08:50:36.766 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-23T08:50:36.767 DEBUG:teuthology.orchestra.run.smithi140:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-23T08:50:36.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-23T08:50:36.871 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-23T08:50:36.871 DEBUG:teuthology.orchestra.run.smithi140:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-23T08:50:36.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-23T08:50:36.928 DEBUG:teuthology.orchestra.run.smithi140:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-23T08:50:36.962 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern = core 2023-09-23T08:50:36.986 DEBUG:teuthology.orchestra.run.smithi140:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-23T08:50:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T08:50:37.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-23T08:50:37.054 INFO:teuthology.task.internal:Transferring archived files... 2023-09-23T08:50:37.055 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/remote/smithi140 2023-09-23T08:50:37.056 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-23T08:50:37.145 INFO:teuthology.task.internal:Removing archive directory... 2023-09-23T08:50:37.145 DEBUG:teuthology.orchestra.run.smithi140:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-23T08:50:37.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-23T08:50:37.174 INFO:teuthology.task.internal:Not uploading archives. 2023-09-23T08:50:37.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-23T08:50:37.201 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-23T08:50:37.202 DEBUG:teuthology.orchestra.run.smithi140:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-23T08:50:37.222 INFO:teuthology.orchestra.run.smithi140.stdout: 393224 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 23 08:50 /home/ubuntu/cephtest 2023-09-23T08:50:37.222 INFO:teuthology.orchestra.run.smithi140.stdout: 393232 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 23 07:12 /home/ubuntu/cephtest/mnt.0 2023-09-23T08:50:37.222 INFO:teuthology.orchestra.run.smithi140.stdout: 393253 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 23 07:14 /home/ubuntu/cephtest/mnt.0/client.0 2023-09-23T08:50:37.222 INFO:teuthology.orchestra.run.smithi140.stdout: 393255 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 23 07:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-23T08:50:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T08:50:37.223 INFO:teuthology.orchestra.run.smithi140.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-09-23T08:50:37.224 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-23T08:50:37.231 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-23T08:50:37.242 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-23T08:50:37.270 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi140 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-23T08:50:37.270 INFO:teuthology.nuke:Checking targets against current locks 2023-09-23T08:50:37.303 DEBUG:teuthology.nuke:shortname: smithi140 2023-09-23T08:50:37.304 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-23T08:50:37.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-23 06:51:25.593899', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM7EfadN9AifUj3gzNG/vZ54NYmDnMRc27pZKNKLHzZBLjbo6PApZOFYWXCd3JzsDiy0T0vb/YQkmWSvAIr63Vw='} 2023-09-23T08:50:37.370 INFO:teuthology.orchestra.console.smithi140:Power off 2023-09-23T08:50:37.371 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-23T08:50:37.394 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off 2023-09-23T08:50:37.394 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T08:50:37.416 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-09-23T08:50:41.418 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T08:50:41.528 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-09-23T08:50:45.530 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T08:50:45.639 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-09-23T08:50:49.640 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T08:50:49.750 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-09-23T08:50:49.851 INFO:teuthology.orchestra.console.smithi140:Power off completed 2023-09-23T08:50:49.957 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} duration: 6604.764305114746 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi140 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5731a07b317e40dc8607ec4c9d013822 status: fail success: false 2023-09-23T08:50:49.957 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-23T08:50:50.032 INFO:teuthology.run:FAIL